By type
*** retrieve Truststore and Keystore password
*** actual launch of AAI Resources
2021-08-11 15:54:17.252 INFO --- [ main] org.onap.aai.util.AAIConfig : Initializing AAIConfig
2021-08-11 15:54:17.260 DEBUG --- [ main] org.onap.aai.util.AAIConfig : Reloading config from /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2021-08-11 15:54:17.260 INFO --- [ main] org.onap.aai.util.AAIConfig : Found the aaiconfig.properties in the following location: /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2021-08-11 15:54:17.262 INFO --- [ main] org.onap.aai.util.AAIConfig : A&AI Server Node Name = aai.config.nodename
2021-08-11 15:54:19.158 WARN 10 --- [ main] o.s.b.l.logback.LogbackLoggingSystem : Ignoring 'logback.configurationFile' system property. Please use 'logging.config' instead.
2021-08-11 15:54:19.448 DEBUG 10 --- [kground-preinit] .h.v.m.ResourceBundleMessageInterpolator : Loaded expression factory via original TCCL
. ____ _ __ _ _
/\\ / ___'_ __ _ _(_)_ __ __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
\\/ ___)| |_)| | | | | || (_| | ) ) ) )
' |____| .__|_| |_|_| |_\__, | / / / /
=========|_|==============|___/=/_/_/_/
:: Spring Boot :: (v2.1.12.RELEASE)
2021-08-11 15:54:19.537 DEBUG 10 --- [kground-preinit] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.messageinterpolation.ResourceBundleMessageInterpolator as ValidatorFactory-scoped message interpolator.
2021-08-11 15:54:19.538 DEBUG 10 --- [kground-preinit] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.resolver.JPATraversableResolver as ValidatorFactory-scoped traversable resolver.
2021-08-11 15:54:19.538 DEBUG 10 --- [kground-preinit] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.util.ExecutableParameterNameProvider as ValidatorFactory-scoped parameter name provider.
2021-08-11 15:54:19.538 DEBUG 10 --- [kground-preinit] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.DefaultClockProvider as ValidatorFactory-scoped clock provider.
2021-08-11 15:54:19.539 DEBUG 10 --- [kground-preinit] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.scripting.DefaultScriptEvaluatorFactory as ValidatorFactory-scoped script evaluator factory.
15:54:25,396 |-INFO in LogbackRequestLog - Will use configuration resource [/localhost-access-logback.xml]
15:54:25,403 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - debug attribute not set
15:54:25,403 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - About to instantiate appender of type [ch.qos.logback.core.rolling.RollingFileAppender]
15:54:25,403 |-INFO in ch.qos.logback.core.joran.action.AppenderAction - Naming appender as [ACCESS]
15:54:25,404 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@1213818572 - No compression will be used
15:54:25,404 |-INFO in c.q.l.core.rolling.TimeBasedRollingPolicy@1213818572 - Will use the pattern /opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log.%d{yyyy-MM-dd} for the active file
15:54:25,404 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - The date pattern is 'yyyy-MM-dd' from file name pattern '/opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log.%d{yyyy-MM-dd}'.
15:54:25,404 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Roll-over at midnight.
15:54:25,405 |-INFO in c.q.l.core.rolling.DefaultTimeBasedFileNamingAndTriggeringPolicy - Setting initial period to Wed Aug 11 15:54:25 GMT 2021
15:54:25,428 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - Active log file name: /opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log
15:54:25,428 |-INFO in ch.qos.logback.core.rolling.RollingFileAppender[ACCESS] - File property is set to [/opt/app/aai-traversal/logs/ajsc-jetty/localhost_access.log]
15:54:25,428 |-INFO in ch.qos.logback.core.joran.action.AppenderRefAction - Attaching appender named [ACCESS] to null
15:54:25,428 |-INFO in ch.qos.logback.access.joran.action.ConfigurationAction - End of configuration.
15:54:25,428 |-INFO in ch.qos.logback.access.joran.JoranConfigurator@1de76cc7 - Registering current configuration as safe fallback point
2021-08-11 15:54:25.610 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED CLASSES [class org.onap.aai.rest.QueryConsumer, class org.onap.aai.rest.CQ2Gremlin, class org.onap.aai.rest.CQ2GremlinTest, class org.onap.aai.rest.search.SearchProvider, class org.onap.aai.rest.util.EchoResponse, class org.onap.aai.rest.search.ModelAndNamedQueryRestProvider, class org.onap.aai.rest.DslConsumer, class org.onap.aai.rest.RecentAPIConsumer]
2021-08-11 15:54:25.614 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestTransactionLogging result=true
2021-08-11 15:54:25.615 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionInterceptor result=true
2021-08-11 15:54:25.615 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseTransactionLogging result=true
2021-08-11 15:54:25.616 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RetiredInterceptor result=true
2021-08-11 15:54:25.616 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.InvalidResponseStatus result=true
2021-08-11 15:54:25.616 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseHeaderManipulation result=true
2021-08-11 15:54:25.617 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HeaderValidation result=true
2021-08-11 15:54:25.617 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestHeaderManipulation result=true
2021-08-11 15:54:25.618 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestModification result=true
2021-08-11 15:54:25.618 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HttpHeaderInterceptor result=true
2021-08-11 15:54:25.618 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.OneWaySslAuthorization result=false
2021-08-11 15:54:25.619 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionLatestInterceptor result=true
2021-08-11 15:54:25.626 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestTransactionLogging result=true
2021-08-11 15:54:25.627 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionInterceptor result=true
2021-08-11 15:54:25.627 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseTransactionLogging result=true
2021-08-11 15:54:25.628 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RetiredInterceptor result=true
2021-08-11 15:54:25.628 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.InvalidResponseStatus result=true
2021-08-11 15:54:25.628 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.post.ResponseHeaderManipulation result=true
2021-08-11 15:54:25.629 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HeaderValidation result=true
2021-08-11 15:54:25.629 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestHeaderManipulation result=true
2021-08-11 15:54:25.630 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.RequestModification result=true
2021-08-11 15:54:25.630 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.HttpHeaderInterceptor result=true
2021-08-11 15:54:25.630 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.OneWaySslAuthorization result=false
2021-08-11 15:54:25.631 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : isEnabledByActiveProfiles: annotatedElement: class org.onap.aai.interceptors.pre.VersionLatestInterceptor result=true
2021-08-11 15:54:25.631 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RequestTransactionLogging
2021-08-11 15:54:25.632 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.post.InvalidResponseStatus
2021-08-11 15:54:25.632 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.HeaderValidation
2021-08-11 15:54:25.633 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.post.ResponseTransactionLogging
2021-08-11 15:54:25.633 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.post.ResponseHeaderManipulation
2021-08-11 15:54:25.633 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.HttpHeaderInterceptor
2021-08-11 15:54:25.634 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.VersionLatestInterceptor
2021-08-11 15:54:25.634 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RetiredInterceptor
2021-08-11 15:54:25.635 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.VersionInterceptor
2021-08-11 15:54:25.635 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RequestHeaderManipulation
2021-08-11 15:54:25.635 DEBUG 10 --- [ main] org.onap.aai.web.JerseyConfiguration : REGISTERED FILTERS org.onap.aai.interceptors.pre.RequestModification
2021-08-11 15:54:26.067 INFO 10 --- [ main] org.onap.aai.config.NodesConfiguration : Translator is SchemaServiceTranslator
2021-08-11 15:54:26.534 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:26.534 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:26.534 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:26.535 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:54:26.547 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/versions, BODY=
2021-08-11 15:54:26.667 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:26.667 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:26.667 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:26.668 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:27.128 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:27.244 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Processing the translator
2021-08-11 15:54:27.244 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv11
2021-08-11 15:54:27.245 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:27.340 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:27.340 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:27.340 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:27.340 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:27.363 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:27.417 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:27.417 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:27.418 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:31.272 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 222
2021-08-11 15:54:31.342 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv12
2021-08-11 15:54:31.342 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:31.345 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:31.345 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:31.345 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:31.345 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:31.368 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:31.469 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:31.470 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:31.471 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:33.318 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 235
2021-08-11 15:54:33.340 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv13
2021-08-11 15:54:33.340 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:33.343 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:33.343 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:33.343 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:33.343 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:33.364 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:33.389 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:33.389 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:33.390 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:34.946 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 237
2021-08-11 15:54:34.967 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv14
2021-08-11 15:54:34.967 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:34.969 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:34.969 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:34.969 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:34.969 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:35.035 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:35.052 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:35.052 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:35.052 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:36.864 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 272
2021-08-11 15:54:36.875 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv15
2021-08-11 15:54:36.876 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:36.878 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:36.878 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:36.879 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:36.879 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:36.926 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:36.944 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:36.944 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:36.945 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:38.609 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 279
2021-08-11 15:54:38.621 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv16
2021-08-11 15:54:38.621 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:38.623 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:38.623 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:38.623 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:38.623 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:38.640 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:38.657 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:38.658 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:38.658 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:40.075 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 281
2021-08-11 15:54:40.086 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv17
2021-08-11 15:54:40.086 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:40.088 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:40.088 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:40.088 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:40.088 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:40.103 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:40.119 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:40.120 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:40.120 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:41.854 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 281
2021-08-11 15:54:41.864 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv18
2021-08-11 15:54:41.864 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:41.866 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:41.867 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:41.867 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:41.867 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:41.883 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:41.905 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:41.906 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:41.906 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:43.475 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 281
2021-08-11 15:54:43.485 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv19
2021-08-11 15:54:43.486 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:43.487 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:43.488 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:43.488 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:43.488 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:43.508 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:43.531 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:43.531 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:43.532 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:45.542 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 301
2021-08-11 15:54:45.552 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv20
2021-08-11 15:54:45.553 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:45.555 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:45.555 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:45.556 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:45.556 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:45.574 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:45.594 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:45.594 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:45.595 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:47.127 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 305
2021-08-11 15:54:47.137 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv21
2021-08-11 15:54:47.138 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:47.140 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:47.140 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:47.140 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:47.141 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:47.162 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:47.184 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:47.185 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:47.185 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:48.943 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 371
2021-08-11 15:54:48.955 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv22
2021-08-11 15:54:48.955 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:48.958 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:48.958 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:48.958 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:48.958 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:48.982 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:49.011 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:49.012 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:49.012 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:51.102 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 378
2021-08-11 15:54:51.121 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Version being processedv23
2021-08-11 15:54:51.121 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Headers: {Accept=application/xml, Content-Type=application/xml}
2021-08-11 15:54:51.125 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:54:51.125 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:54:51.125 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:54:51.125 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:54:51.186 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:54:51.221 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:54:51.222 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Retrieved OXMs from SchemaService
2021-08-11 15:54:51.222 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Ingested the InputStream
2021-08-11 15:54:53.752 DEBUG 10 --- [ main] org.onap.aai.nodes.NodeIngestor : Types size 378
2021-08-11 15:54:53.773 DEBUG 10 --- [ main] org.onap.aai.TraversalApp : AAI Server initialization started...
2021-08-11 15:54:53.774 DEBUG 10 --- [ main] org.onap.aai.TraversalApp : Starting AAIGraph connections and the NodeInjestor
2021-08-11 15:54:53.774 INFO 10 --- [ main] org.onap.aai.util.AAIConfig : Initializing AAIConfig
2021-08-11 15:54:53.774 DEBUG 10 --- [ main] org.onap.aai.util.AAIConfig : Reloading config from /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2021-08-11 15:54:53.774 INFO 10 --- [ main] org.onap.aai.util.AAIConfig : Found the aaiconfig.properties in the following location: /opt/app/aai-traversal/./resources/etc/appprops/aaiconfig.properties
2021-08-11 15:54:53.775 INFO 10 --- [ main] org.onap.aai.util.AAIConfig : A&AI Server Node Name = aai.config.nodename
2021-08-11 15:54:54.310 INFO 10 --- [ main] com.datastax.driver.core : DataStax Java driver 3.6.0 for Apache Cassandra
2021-08-11 15:54:54.320 INFO 10 --- [ main] c.d.driver.core.GuavaCompatibility : Detected Guava >= 19 in the classpath, using modern compatibility layer
2021-08-11 15:54:54.321 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.NEW_NODE_DELAY_SECONDS is undefined, using default value 1
2021-08-11 15:54:54.321 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.NOTIF_LOCK_TIMEOUT_SECONDS is undefined, using default value 60
2021-08-11 15:54:54.340 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.USE_NATIVE_CLOCK is undefined, using default value true
2021-08-11 15:54:54.634 INFO 10 --- [ main] com.datastax.driver.core.ClockFactory : Using native clock to generate timestamps.
2021-08-11 15:54:54.645 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.NON_BLOCKING_EXECUTOR_SIZE is undefined, using default value 2
2021-08-11 15:54:54.740 DEBUG 10 --- [ main] com.datastax.driver.core.Cluster : Starting new cluster with contact points [cassandra/10.242.110.137:9042]
2021-08-11 15:54:54.783 DEBUG 10 --- [ main] i.n.u.i.logging.InternalLoggerFactory : Using SLF4J as the default logging framework
2021-08-11 15:54:54.809 DEBUG 10 --- [ main] i.n.u.internal.InternalThreadLocalMap : -Dio.netty.threadLocalMap.stringBuilder.initialSize: 1024
2021-08-11 15:54:54.809 DEBUG 10 --- [ main] i.n.u.internal.InternalThreadLocalMap : -Dio.netty.threadLocalMap.stringBuilder.maxSize: 4096
2021-08-11 15:54:54.861 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : -Dio.netty.noUnsafe: false
2021-08-11 15:54:54.861 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : Java version: 8
2021-08-11 15:54:54.863 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : sun.misc.Unsafe.theUnsafe: available
2021-08-11 15:54:54.864 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : sun.misc.Unsafe.copyMemory: available
2021-08-11 15:54:54.865 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : java.nio.Buffer.address: available
2021-08-11 15:54:54.866 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : direct buffer constructor: available
2021-08-11 15:54:54.868 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : java.nio.Bits.unaligned: available, true
2021-08-11 15:54:54.868 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : jdk.internal.misc.Unsafe.allocateUninitializedArray(int): unavailable prior to Java9
2021-08-11 15:54:54.868 DEBUG 10 --- [ main] i.n.util.internal.PlatformDependent0 : java.nio.DirectByteBuffer.(long, int): available
2021-08-11 15:54:54.868 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : sun.misc.Unsafe: available
2021-08-11 15:54:54.868 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.tmpdir: /tmp (java.io.tmpdir)
2021-08-11 15:54:54.868 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.bitMode: 64 (sun.arch.data.model)
2021-08-11 15:54:54.871 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.maxDirectMemory: 1020067840 bytes
2021-08-11 15:54:54.871 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.uninitializedArrayAllocationThreshold: -1
2021-08-11 15:54:54.873 DEBUG 10 --- [ main] io.netty.util.internal.CleanerJava6 : java.nio.ByteBuffer.cleaner(): available
2021-08-11 15:54:54.873 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : -Dio.netty.noPreferDirect: false
2021-08-11 15:54:54.879 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.FORCE_NIO is undefined, using default value false
2021-08-11 15:54:54.887 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : -Dio.netty.native.workdir: /tmp (io.netty.tmpdir)
2021-08-11 15:54:54.887 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : -Dio.netty.native.deleteLibAfterLoading: true
2021-08-11 15:54:54.887 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : -Dio.netty.native.tryPatchShadedId: true
2021-08-11 15:54:54.892 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : Unable to load the library 'netty_transport_native_epoll_x86_64', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:287)
at io.netty.channel.epoll.Native.(Native.java:69)
at io.netty.channel.epoll.Epoll.(Epoll.java:33)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
2021-08-11 15:54:54.893 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : netty_transport_native_epoll_x86_64 cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:287)
at io.netty.channel.epoll.Native.(Native.java:69)
at io.netty.channel.epoll.Epoll.(Epoll.java:33)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
2021-08-11 15:54:54.919 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : Unable to load the library '/tmp/libnetty_transport_native_epoll_x86_644238516212855909297.so', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: null
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:382)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:203)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:287)
at io.netty.channel.epoll.Native.(Native.java:69)
at io.netty.channel.epoll.Epoll.(Epoll.java:33)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
... 65 common frames omitted
Caused by: java.lang.RuntimeException: failed to get field ID: DefaultFileRegion.transfered
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
... 72 common frames omitted
2021-08-11 15:54:54.920 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : Unable to load the library 'netty_transport_native_epoll', trying other loading mechanism.
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:290)
at io.netty.channel.epoll.Native.(Native.java:69)
at io.netty.channel.epoll.Epoll.(Epoll.java:33)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
2021-08-11 15:54:54.921 DEBUG 10 --- [ main] i.n.util.internal.NativeLibraryLoader : netty_transport_native_epoll cannot be loaded from java.library.path, now trying export to -Dio.netty.native.workdir: /tmp
java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:290)
at io.netty.channel.epoll.Native.(Native.java:69)
at io.netty.channel.epoll.Epoll.(Epoll.java:33)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
2021-08-11 15:54:54.923 WARN 10 --- [ main] com.datastax.driver.core.NettyUtil : Found Netty's native epoll transport in the classpath, but epoll is not available. Using NIO instead.
java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll_x86_64
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:287)
at io.netty.channel.epoll.Native.(Native.java:69)
at io.netty.channel.epoll.Epoll.(Epoll.java:33)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:264)
at com.datastax.driver.core.NettyUtil.(NettyUtil.java:57)
at com.datastax.driver.core.NettyOptions.eventLoopGroup(NettyOptions.java:95)
at com.datastax.driver.core.Connection$Factory.(Connection.java:926)
at com.datastax.driver.core.Cluster$Manager.init(Cluster.java:1499)
at com.datastax.driver.core.Cluster.init(Cluster.java:208)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:376)
at com.datastax.driver.core.Cluster.connectAsync(Cluster.java:355)
at com.datastax.driver.core.Cluster.connect(Cluster.java:305)
at org.janusgraph.diskstorage.cql.CQLStoreManager.initializeSession(CQLStoreManager.java:272)
at org.janusgraph.diskstorage.cql.CQLStoreManager.(CQLStoreManager.java:165)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at org.janusgraph.util.system.ConfigurationUtil.instantiate(ConfigurationUtil.java:58)
at org.janusgraph.diskstorage.Backend.getImplementationClass(Backend.java:477)
at org.janusgraph.diskstorage.Backend.getStorageManager(Backend.java:409)
at org.janusgraph.graphdb.configuration.GraphDatabaseConfiguration.(GraphDatabaseConfiguration.java:1382)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:164)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:133)
at org.janusgraph.core.JanusGraphFactory.open(JanusGraphFactory.java:113)
at org.onap.aai.dbmap.AAIGraph.loadGraph(AAIGraph.java:107)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:73)
at org.onap.aai.dbmap.AAIGraph.(AAIGraph.java:54)
at org.onap.aai.dbmap.AAIGraph$Helper.(AAIGraph.java:80)
at org.onap.aai.dbmap.AAIGraph.getInstance(AAIGraph.java:94)
at org.onap.aai.TraversalApp.init(TraversalApp.java:114)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleElement.invoke(InitDestroyAnnotationBeanPostProcessor.java:363)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor$LifecycleMetadata.invokeInitMethods(InitDestroyAnnotationBeanPostProcessor.java:307)
at org.springframework.beans.factory.annotation.InitDestroyAnnotationBeanPostProcessor.postProcessBeforeInitialization(InitDestroyAnnotationBeanPostProcessor.java:136)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.applyBeanPostProcessorsBeforeInitialization(AbstractAutowireCapableBeanFactory.java:414)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1763)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:593)
at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:515)
at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:320)
at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:318)
at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:847)
at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:877)
at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:549)
at org.springframework.boot.web.servlet.context.ServletWebServerApplicationContext.refresh(ServletWebServerApplicationContext.java:141)
at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:744)
at org.springframework.boot.SpringApplication.refreshContext(SpringApplication.java:391)
at org.springframework.boot.SpringApplication.run(SpringApplication.java:312)
at org.onap.aai.TraversalApp.main(TraversalApp.java:136)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:87)
at org.springframework.boot.loader.Launcher.launch(Launcher.java:51)
at org.springframework.boot.loader.PropertiesLauncher.main(PropertiesLauncher.java:578)
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
... 63 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll_x86_64 in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: could not load a native library: netty_transport_native_epoll
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:224)
at io.netty.channel.epoll.Native.loadNativeLibrary(Native.java:290)
... 62 common frames omitted
Caused by: java.io.FileNotFoundException: META-INF/native/libnetty_transport_native_epoll.so
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:173)
... 63 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:136)
... 63 common frames omitted
Suppressed: java.lang.UnsatisfiedLinkError: no netty_transport_native_epoll in java.library.path
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1867)
at java.lang.Runtime.loadLibrary0(Runtime.java:870)
at java.lang.System.loadLibrary(System.java:1122)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:38)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at io.netty.util.internal.NativeLibraryLoader$1.run(NativeLibraryLoader.java:369)
at java.security.AccessController.doPrivileged(Native Method)
at io.netty.util.internal.NativeLibraryLoader.loadLibraryByHelper(NativeLibraryLoader.java:361)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:339)
... 64 common frames omitted
Caused by: java.lang.RuntimeException: failed to get field ID: DefaultFileRegion.transfered
at java.lang.ClassLoader$NativeLibrary.load(Native Method)
at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1941)
at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1824)
at java.lang.Runtime.load0(Runtime.java:809)
at java.lang.System.load(System.java:1086)
at io.netty.util.internal.NativeLibraryUtil.loadLibrary(NativeLibraryUtil.java:36)
at io.netty.util.internal.NativeLibraryLoader.loadLibrary(NativeLibraryLoader.java:349)
at io.netty.util.internal.NativeLibraryLoader.load(NativeLibraryLoader.java:203)
... 63 common frames omitted
2021-08-11 15:54:54.934 DEBUG 10 --- [ main] i.n.channel.MultithreadEventLoopGroup : -Dio.netty.eventLoopThreads: 4
2021-08-11 15:54:54.952 DEBUG 10 --- [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.noKeySetOptimization: false
2021-08-11 15:54:54.952 DEBUG 10 --- [ main] io.netty.channel.nio.NioEventLoop : -Dio.netty.selectorAutoRebuildThreshold: 512
2021-08-11 15:54:54.972 DEBUG 10 --- [ main] i.netty.util.internal.PlatformDependent : org.jctools-core.MpscChunkedArrayQueue: available
2021-08-11 15:54:55.007 DEBUG 10 --- [ main] io.netty.util.ResourceLeakDetector : -Dio.netty.leakDetection.level: simple
2021-08-11 15:54:55.008 DEBUG 10 --- [ main] io.netty.util.ResourceLeakDetector : -Dio.netty.leakDetection.targetRecords: 4
2021-08-11 15:54:55.009 DEBUG 10 --- [ main] i.n.util.ResourceLeakDetectorFactory : Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@8318808
2021-08-11 15:54:55.018 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.EXTENDED_PEER_CHECK is undefined, using default value true
2021-08-11 15:54:55.079 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 15:54:55.084 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.DISABLE_COALESCING is undefined, using default value false
2021-08-11 15:54:55.084 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.FLUSHER_SCHEDULE_PERIOD_NS is undefined, using default value 10000
2021-08-11 15:54:55.085 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.FLUSHER_RUN_WITHOUT_WORK_TIMES is undefined, using default value 5
2021-08-11 15:54:55.151 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.numHeapArenas: 4
2021-08-11 15:54:55.151 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.numDirectArenas: 4
2021-08-11 15:54:55.151 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.pageSize: 8192
2021-08-11 15:54:55.152 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.maxOrder: 11
2021-08-11 15:54:55.152 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.chunkSize: 16777216
2021-08-11 15:54:55.152 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.tinyCacheSize: 512
2021-08-11 15:54:55.152 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.smallCacheSize: 256
2021-08-11 15:54:55.152 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.normalCacheSize: 64
2021-08-11 15:54:55.152 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.maxCachedBufferCapacity: 32768
2021-08-11 15:54:55.152 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.cacheTrimInterval: 8192
2021-08-11 15:54:55.153 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.cacheTrimIntervalMillis: 0
2021-08-11 15:54:55.153 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.useCacheForAllThreads: true
2021-08-11 15:54:55.153 DEBUG 10 --- [ main] io.netty.buffer.PooledByteBufAllocator : -Dio.netty.allocator.maxCachedByteBuffersPerChunk: 1023
2021-08-11 15:54:55.198 DEBUG 10 --- [ main] io.netty.channel.DefaultChannelId : -Dio.netty.processId: 10 (auto-detected)
2021-08-11 15:54:55.202 DEBUG 10 --- [ main] io.netty.util.NetUtil : -Djava.net.preferIPv4Stack: false
2021-08-11 15:54:55.202 DEBUG 10 --- [ main] io.netty.util.NetUtil : -Djava.net.preferIPv6Addresses: false
2021-08-11 15:54:55.204 DEBUG 10 --- [ main] io.netty.util.NetUtil : Loopback interface: lo (lo, 0:0:0:0:0:0:0:1%lo)
2021-08-11 15:54:55.206 DEBUG 10 --- [ main] io.netty.util.NetUtil : /proc/sys/net/core/somaxconn: 128
2021-08-11 15:54:55.207 DEBUG 10 --- [ main] io.netty.channel.DefaultChannelId : -Dio.netty.machineId: e6:d1:11:ff:fe:f6:47:0a (auto-detected)
2021-08-11 15:54:55.256 DEBUG 10 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.allocator.type: pooled
2021-08-11 15:54:55.256 DEBUG 10 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.threadLocalDirectBufferSize: 0
2021-08-11 15:54:55.256 DEBUG 10 --- [ main] io.netty.buffer.ByteBufUtil : -Dio.netty.maxThreadLocalCharBufferSize: 16384
2021-08-11 15:54:55.338 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:54:55.386 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.maxCapacityPerThread: 4096
2021-08-11 15:54:55.386 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.maxSharedCapacityFactor: 2
2021-08-11 15:54:55.386 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.linkCapacity: 16
2021-08-11 15:54:55.386 DEBUG 10 --- [er-nio-worker-0] io.netty.util.Recycler : -Dio.netty.recycler.ratio: 8
2021-08-11 15:54:55.400 DEBUG 10 --- [er-nio-worker-0] io.netty.buffer.AbstractByteBuf : -Dio.netty.buffer.checkAccessible: true
2021-08-11 15:54:55.400 DEBUG 10 --- [er-nio-worker-0] io.netty.buffer.AbstractByteBuf : -Dio.netty.buffer.checkBounds: true
2021-08-11 15:54:55.400 DEBUG 10 --- [er-nio-worker-0] i.n.util.ResourceLeakDetectorFactory : Loaded default ResourceLeakDetector: io.netty.util.ResourceLeakDetector@b1c4019
2021-08-11 15:54:55.484 DEBUG 10 --- [er-nio-worker-0] c.datastax.driver.core.SystemProperties : com.datastax.driver.NATIVE_TRANSPORT_MAX_FRAME_SIZE_IN_MB is undefined, using default value 256
2021-08-11 15:54:56.561 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:54:56.563 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2021-08-11 15:54:56.676 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2021-08-11 15:54:56.932 DEBUG 10 --- [ main] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:54:56.942 DEBUG 10 --- [ main] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 10 milliseconds
2021-08-11 15:54:56.956 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [Control connection] established to cassandra/10.242.110.137:9042
2021-08-11 15:54:56.956 INFO 10 --- [ main] c.d.d.c.p.DCAwareRoundRobinPolicy : Using data-center name 'Pod' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2021-08-11 15:54:56.958 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.242.184.131:9042 added
2021-08-11 15:54:56.958 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.242.154.72:9042 added
2021-08-11 15:54:56.958 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host cassandra/10.242.110.137:9042 added
2021-08-11 15:54:56.963 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.CHECK_IO_DEADLOCKS is undefined, using default value true
2021-08-11 15:54:56.968 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 15:54:56.972 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:54:56.976 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 1
2021-08-11 15:54:56.976 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 2
2021-08-11 15:54:56.977 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:54:56.979 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:54:57.252 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:54:57.255 DEBUG 10 --- [er-nio-worker-0] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.154.72:9042 (1 connections needed, 1 successfully opened)
2021-08-11 15:54:57.255 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Session : Added connection pool for /10.242.154.72:9042
2021-08-11 15:54:57.665 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:54:57.666 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 15:54:57.666 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 15:54:58.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:54:58.362 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 15:54:58.362 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 15:54:58.752 DEBUG 10 --- [ main] org.reflections.Reflections : going to scan these urls:
file:/opt/app/aai-traversal/resources/
2021-08-11 15:54:58.763 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application-keycloak.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.763 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application-keycloak.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.765 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ipsec-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.765 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ipsec-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.765 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.765 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/reserved-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/reserved-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tagged-inventory-item-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tagged-inventory-item-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.766 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/constrained-element-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/constrained-element-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tenant-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tenant-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.767 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vf-module-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vf-module-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/newvce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/newvce-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.768 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/customer-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/customer-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vig-server-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vig-server-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnf-image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnf-image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/snapshot-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.769 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/snapshot-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv4-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv4-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/oam-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.770 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/oam-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/entitlement-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/entitlement-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/lag-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.771 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/collection-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/collection-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/result-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/result-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/allotted-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/allotted-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/group-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.772 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/group-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-key-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-key-resource-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/update-node-key-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/update-node-key-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cloud-region-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.773 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cloud-region-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/route-table-reference-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/route-table-reference-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cvlan-tag-entry-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/cvlan-tag-entry-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.774 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-capability-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-capability-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/pserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/connector-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/connector-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/segmentation-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.775 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/segmentation-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/property-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/property-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/routing-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/routing-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.776 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/related-lookup-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/related-lookup-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/class-of-service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/class-of-service-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/key-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.777 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/key-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-policy-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-policy-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/license-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/logical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.778 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/logical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpn-binding-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpn-binding-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/instance-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.779 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/instance-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/start-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/start-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-constraint-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.780 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/image-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-profile-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/network-profile-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-assignment-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/port-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.781 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/port-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/multicast-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/multicast-configuration-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/p-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/p-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/physical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/physical-link-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.782 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/generic-vnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/generic-vnf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l-interface-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/az-and-dvs-switches-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/az-and-dvs-switches-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/virtual-data-center-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.783 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/virtual-data-center-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv6-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-interface-ipv6-address-list-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/secondary-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/secondary-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vlan-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vlan-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.784 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/site-pair-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/edge-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/edge-prop-names-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/metadatum-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/metadatum-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/availability-zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/availability-zone-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.785 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/dvs-switch-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/dvs-switch-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/action-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.786 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/named-query-element-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/subnet-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/subnet-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/include-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/include-node-filter-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnfc-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vnfc-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.787 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/flavor-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/flavor-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/volume-group-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/complex-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/complex-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-pool-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/ctag-pool-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.788 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/element-choice-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/element-choice-set-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/l3-network-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpls-pe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/vpls-pe-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-subscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.789 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/service-subscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/inventory-item-data-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/sriov-vf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/sriov-vf-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tunnel-xconnect-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/tunnel-xconnect-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.790 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-ver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/widget-model-json/model-ver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/service-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/service-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/resource-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.791 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/resource-model-json/000-README in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology2-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology2-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.792 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVserver-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/closed-loop-named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/closed-loop-named-query-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcInstance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcInstance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.793 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVServiceTopology-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcSubscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getTenantInfoAtSvcSubscription-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.2.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.2.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceModel-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getServiceInstanceModel-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getWlBundleId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.794 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getWlBundleId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLinkByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLinkByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexLocationId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexLocationId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getRouterRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getRouterRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-service-instance-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.795 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getLogicalLinkByCloudRegionId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getLogicalLinkByCloudRegionId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.1.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getSvcSubscriberModelInfo-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getSvcSubscriberModelInfo-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getL3networkCloudRegionByNetworkRole-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getL3networkCloudRegionByNetworkRole-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.796 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getClfiRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getClfiRoadmTailSummary-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVnfVlanByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getVnfVlanByCircuitId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexHostname-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVComplexHostname-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getPinterfacePhysicalLinkBySvcInstId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getPinterfacePhysicalLinkBySvcInstId-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.797 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLink-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getDHVLogicalLink-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/vnf-to-esr-system-info-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComponentList-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComplexByPnfName-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.798 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/scriptdata/named-query-json/getComplexByPnfName-1.0.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/aai_keystore in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/aai_keystore in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/realm.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/auth/realm.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.799 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/logging.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/logging.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiEventDMaaPPublisher.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiEventDMaaPPublisher.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-realtime.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-realtime.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.800 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-cached.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.801 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/janusgraph-cached.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.801 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/error.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.801 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/error.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.801 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiconfig.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.801 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file etc/appprops/aaiconfig.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/query/stored-queries.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/query/stored-queries.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v9/aai_oxm_v9.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v9/aai_oxm_v9.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v14/aai_oxm_v14.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.802 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v14/aai_oxm_v14.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.803 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v19/aai_oxm_v19.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.803 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v19/aai_oxm_v19.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.803 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v15/aai_oxm_v15.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v15/aai_oxm_v15.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v21/aai_oxm_v21.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v21/aai_oxm_v21.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v17/aai_oxm_v17.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.804 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v17/aai_oxm_v17.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.805 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v13/aai_oxm_v13.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.805 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v13/aai_oxm_v13.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.805 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v8/aai_oxm_v8.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.805 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v8/aai_oxm_v8.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.806 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v18/aai_oxm_v18.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.806 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v18/aai_oxm_v18.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.806 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v11/aai_oxm_v11.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.806 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v11/aai_oxm_v11.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.807 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v22/aai_oxm_v22.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.807 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v22/aai_oxm_v22.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.807 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v23/aai_oxm_v23.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.807 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v23/aai_oxm_v23.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v16/aai_oxm_v16.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v16/aai_oxm_v16.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v20/aai_oxm_v20.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v20/aai_oxm_v20.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.808 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v10/aai_oxm_v10.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v10/aai_oxm_v10.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v12/aai_oxm_v12.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/oxm/v12/aai_oxm_v12.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v9/DbEdgeRules_v9.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.809 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v9/DbEdgeRules_v9.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_esr_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_esr_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_hpa_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_hpa_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.810 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_ccvpn_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_ccvpn_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_pnp_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v14/DbEdgeRules_pnp_v14.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.811 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_esr_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_esr_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_hpa_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_hpa_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_ccvpn_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.812 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_ccvpn_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_bbs_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_bbs_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_pnp_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v19/DbEdgeRules_pnp_v19.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_hpa_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_hpa_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.813 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_pnp_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_pnp_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_ccvpn_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_ccvpn_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_esr_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v15/DbEdgeRules_esr_v15.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.814 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_esr_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_esr_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_hpa_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_hpa_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_pnp_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.815 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_pnp_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_ccvpn_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_ccvpn_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_bbs_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v21/DbEdgeRules_bbs_v21.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_hpa_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.816 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_hpa_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_ccvpn_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_ccvpn_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_bbs_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_bbs_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_pnp_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_pnp_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.817 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_esr_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_esr_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v17/DbEdgeRules_v17.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_esr_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_esr_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.818 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_hpa_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v13/DbEdgeRules_hpa_v13.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v8/DbEdgeRules_v8.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v8/DbEdgeRules_v8.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_hpa_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_hpa_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.819 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_ccvpn_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_ccvpn_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_pnp_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_pnp_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_bbs_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_bbs_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_esr_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.820 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_esr_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v18/DbEdgeRules_v18.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_esr_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_esr_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v11/DbEdgeRules_v11.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.821 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_pnp_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_pnp_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_ccvpn_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_ccvpn_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_bbs_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_bbs_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.822 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_hpa_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_hpa_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_esr_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v22/DbEdgeRules_esr_v22.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_ccvpn_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_ccvpn_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_hpa_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.823 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_hpa_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_bbs_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_bbs_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_pnp_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_pnp_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.824 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_esr_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v23/DbEdgeRules_esr_v23.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_pnp_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_pnp_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_ccvpn_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_ccvpn_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.825 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_hpa_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_hpa_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_esr_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_esr_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_bbs_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v16/DbEdgeRules_bbs_v16.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_bbs_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.826 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_bbs_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_ccvpn_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_ccvpn_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_hpa_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_hpa_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_pnp_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_pnp_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_esr_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.827 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_esr_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v20/DbEdgeRules_v20.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v10/DbEdgeRules_v10.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v10/DbEdgeRules_v10.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_hpa_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.828 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_hpa_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_esr_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file schema/onap/dbedgerules/v12/DbEdgeRules_esr_v12.json in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.829 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/bath_config.csv in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/bath_config.csv in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/permissions.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/permissions.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.osaaf.location.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.830 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/onap/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.keyfile in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.831 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.p12 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/org.onap.aai.props in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file aaf/truststoreONAPall.jks in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file localhost-access-logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.832 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file localhost-access-logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.833 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.833 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.833 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v1/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.833 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v1/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v2/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file antlr4/org/onap/aai/dsl/v2/AAIDsl.g4 in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file logback.xml in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file retired.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file retired.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file cadi.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.834 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file cadi.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.835 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application.properties in url file:/opt/app/aai-traversal/resources/ with scanner TypeAnnotationsScanner
2021-08-11 15:54:58.835 DEBUG 10 --- [ main] org.reflections.Reflections : could not scan file application.properties in url file:/opt/app/aai-traversal/resources/ with scanner SubTypesScanner
2021-08-11 15:54:58.836 INFO 10 --- [ main] org.reflections.Reflections : Reflections took 82 ms to scan 1 urls, producing 0 keys and 0 values
2021-08-11 15:54:59.150 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=true] closing connection
2021-08-11 15:54:59.150 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-1, inFlight=0, closed=true] closed, remaining = 0
2021-08-11 15:54:59.154 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=true] closing connection
2021-08-11 15:54:59.154 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-1, inFlight=0, closed=true] closed, remaining = 0
2021-08-11 15:54:59.154 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=true] closing connection
2021-08-11 15:54:59.154 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=true] closed, remaining = 1
2021-08-11 15:54:59.156 DEBUG 10 --- [ main] com.datastax.driver.core.Cluster : Shutting down
2021-08-11 15:54:59.158 DEBUG 10 --- [ main] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=true] closing connection
2021-08-11 15:54:59.159 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=true] closed, remaining = 0
2021-08-11 15:55:01.172 DEBUG 10 --- [er-nio-worker-1] io.netty.buffer.PoolThreadCache : Freed 20 thread-local buffer(s) from thread: JanusGraph Cluster-nio-worker-1
2021-08-11 15:55:01.172 DEBUG 10 --- [er-nio-worker-0] io.netty.buffer.PoolThreadCache : Freed 38 thread-local buffer(s) from thread: JanusGraph Cluster-nio-worker-0
2021-08-11 15:55:01.201 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.USE_NATIVE_CLOCK is undefined, using default value true
2021-08-11 15:55:01.201 INFO 10 --- [ main] com.datastax.driver.core.ClockFactory : Using native clock to generate timestamps.
2021-08-11 15:55:01.202 DEBUG 10 --- [ main] com.datastax.driver.core.Cluster : Starting new cluster with contact points [cassandra/10.242.110.137:9042]
2021-08-11 15:55:01.207 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 15:55:01.210 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:55:01.870 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:55:01.871 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2021-08-11 15:55:01.879 DEBUG 10 --- [ main] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2021-08-11 15:55:01.994 DEBUG 10 --- [ main] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:55:02.001 DEBUG 10 --- [ main] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 7 milliseconds
2021-08-11 15:55:02.012 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [Control connection] established to cassandra/10.242.110.137:9042
2021-08-11 15:55:02.012 INFO 10 --- [ main] c.d.d.c.p.DCAwareRoundRobinPolicy : Using data-center name 'Pod' for DCAwareRoundRobinPolicy (if this is incorrect, please provide the correct datacenter name with DCAwareRoundRobinPolicy constructor)
2021-08-11 15:55:02.012 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.242.184.131:9042 added
2021-08-11 15:55:02.012 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host /10.242.154.72:9042 added
2021-08-11 15:55:02.012 INFO 10 --- [ main] com.datastax.driver.core.Cluster : New Cassandra host cassandra/10.242.110.137:9042 added
2021-08-11 15:55:02.012 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 15:55:02.013 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 1
2021-08-11 15:55:02.014 DEBUG 10 --- [ main] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 2
2021-08-11 15:55:02.015 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:55:02.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:55:02.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 15:55:02.152 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:55:02.153 DEBUG 10 --- [er-nio-worker-0] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.154.72:9042 (1 connections needed, 1 successfully opened)
2021-08-11 15:55:02.153 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Session : Added connection pool for /10.242.154.72:9042
2021-08-11 15:55:02.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:55:02.156 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 15:55:02.156 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 15:55:02.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 15:55:02.492 DEBUG 10 --- [er-nio-worker-1] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 15:55:02.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 15:55:03.151 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Processing the translator
2021-08-11 15:55:03.153 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv11
2021-08-11 15:55:03.154 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.154 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v11, BODY=
2021-08-11 15:55:03.156 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.156 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.156 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.156 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.178 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.180 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.180 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.181 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv12
2021-08-11 15:55:03.181 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.181 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v12, BODY=
2021-08-11 15:55:03.182 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.182 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.182 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.182 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.192 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.194 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.194 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.194 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv13
2021-08-11 15:55:03.195 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.195 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v13, BODY=
2021-08-11 15:55:03.196 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.196 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.196 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.196 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.207 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.209 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.209 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.209 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv14
2021-08-11 15:55:03.209 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.210 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v14, BODY=
2021-08-11 15:55:03.211 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.211 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.211 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.211 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.225 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.227 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.228 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.228 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv15
2021-08-11 15:55:03.228 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.228 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v15, BODY=
2021-08-11 15:55:03.229 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.230 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.230 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.230 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.240 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.244 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.244 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.244 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv16
2021-08-11 15:55:03.244 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.245 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v16, BODY=
2021-08-11 15:55:03.246 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.246 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.246 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.246 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.260 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.262 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.262 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.263 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv17
2021-08-11 15:55:03.263 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.263 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v17, BODY=
2021-08-11 15:55:03.264 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.264 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.264 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.264 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.275 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.277 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.278 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.278 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv18
2021-08-11 15:55:03.278 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.279 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v18, BODY=
2021-08-11 15:55:03.280 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.280 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.280 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.280 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.294 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.297 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.297 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.297 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv19
2021-08-11 15:55:03.297 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.298 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v19, BODY=
2021-08-11 15:55:03.299 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.299 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.299 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.299 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.311 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.314 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.314 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.314 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv20
2021-08-11 15:55:03.315 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.315 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v20, BODY=
2021-08-11 15:55:03.316 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.316 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.316 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.316 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.328 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.332 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.333 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.333 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv21
2021-08-11 15:55:03.333 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.334 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v21, BODY=
2021-08-11 15:55:03.335 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.335 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.335 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.335 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.347 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.352 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.353 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.353 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv22
2021-08-11 15:55:03.353 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.354 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v22, BODY=
2021-08-11 15:55:03.355 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.355 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.355 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.355 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.369 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.374 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.375 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.375 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Version being processedv23
2021-08-11 15:55:03.375 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.376 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/edgerules?version=v23, BODY=
2021-08-11 15:55:03.377 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.377 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.377 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.377 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.389 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:03.394 DEBUG 10 --- [ main] o.o.aai.setup.SchemaServiceTranslator : SchemaResponse Status code200 OK
2021-08-11 15:55:03.394 DEBUG 10 --- [ main] org.onap.aai.edges.EdgeIngestor : Retrieved json from SchemaService
2021-08-11 15:55:03.517 INFO 10 --- [ main] o.o.a.rest.search.SchemaServiceCQConfig : Calling the SchemaService to retrieve stored queries
2021-08-11 15:55:03.518 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : Request Headers: {}
2021-08-11 15:55:03.519 DEBUG 10 --- [ main] org.onap.aai.restclient.RestClient : METHOD=GET, URL=https://aai-schema-service.onap:8452/aai/schema-service/v1/stored-queries, BODY=
2021-08-11 15:55:03.520 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Checking for system property [partnerName]
2021-08-11 15:55:03.520 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : System property was null or empty. Checking environment variable for: partnerName
2021-08-11 15:55:03.521 INFO 10 --- [ main] org.onap.logging.filter.base.MDCSetup : Environment variable: partnerName was null or empty
2021-08-11 15:55:03.521 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : Invoke
2021-08-11 15:55:03.542 INFO 10 --- [ main] o.o.l.f.base.AbstractMetricLogFilter : InvokeReturn
2021-08-11 15:55:04.185 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Found javax.persistence.Persistence on classpath containing 'getPersistenceUtil'. Assuming JPA 2 environment. Trying to instantiate JPA aware TraversableResolver
2021-08-11 15:55:04.185 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Instantiated JPA aware TraversableResolver of type org.hibernate.validator.internal.engine.resolver.JPATraversableResolver.
2021-08-11 15:55:04.185 DEBUG 10 --- [ main] .h.v.m.ResourceBundleMessageInterpolator : Loaded expression factory via original TCCL
2021-08-11 15:55:04.187 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Found javax.persistence.Persistence on classpath containing 'getPersistenceUtil'. Assuming JPA 2 environment. Trying to instantiate JPA aware TraversableResolver
2021-08-11 15:55:04.187 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Instantiated JPA aware TraversableResolver of type org.hibernate.validator.internal.engine.resolver.JPATraversableResolver.
2021-08-11 15:55:04.189 DEBUG 10 --- [ main] o.h.v.internal.engine.ConfigurationImpl : Setting custom MessageInterpolator of type org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator
2021-08-11 15:55:04.189 DEBUG 10 --- [ main] o.h.v.internal.engine.ConfigurationImpl : Setting custom ConstraintValidatorFactory of type org.springframework.validation.beanvalidation.SpringConstraintValidatorFactory
2021-08-11 15:55:04.190 DEBUG 10 --- [ main] o.h.v.internal.engine.ConfigurationImpl : Setting custom ParameterNameProvider of type org.springframework.validation.beanvalidation.LocalValidatorFactoryBean$1
2021-08-11 15:55:04.193 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : Trying to load META-INF/validation.xml for XML based Validator configuration.
2021-08-11 15:55:04.193 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via user class loader
2021-08-11 15:55:04.193 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via TCCL
2021-08-11 15:55:04.194 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via Hibernate Validator's class loader
2021-08-11 15:55:04.194 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : No META-INF/validation.xml found. Using annotation based configuration only.
2021-08-11 15:55:04.209 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.springframework.validation.beanvalidation.LocaleContextMessageInterpolator as ValidatorFactory-scoped message interpolator.
2021-08-11 15:55:04.209 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.resolver.JPATraversableResolver as ValidatorFactory-scoped traversable resolver.
2021-08-11 15:55:04.209 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.util.ExecutableParameterNameProvider as ValidatorFactory-scoped parameter name provider.
2021-08-11 15:55:04.209 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.DefaultClockProvider as ValidatorFactory-scoped clock provider.
2021-08-11 15:55:04.209 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.scripting.DefaultScriptEvaluatorFactory as ValidatorFactory-scoped script evaluator factory.
2021-08-11 15:55:04.768 DEBUG 10 --- [ main] c.datastax.driver.core.SystemProperties : com.datastax.driver.USE_NATIVE_CLOCK is undefined, using default value true
2021-08-11 15:55:04.768 INFO 10 --- [ main] com.datastax.driver.core.ClockFactory : Using native clock to generate timestamps.
2021-08-11 15:55:04.777 WARN 10 --- [ main] .b.a.g.t.GroovyTemplateAutoConfiguration : Cannot find template location: classpath:/templates/ (please add some templates, check your Groovy configuration, or set spring.groovy.template.check-template-location=false)
2021-08-11 15:55:07.160 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Found javax.persistence.Persistence on classpath containing 'getPersistenceUtil'. Assuming JPA 2 environment. Trying to instantiate JPA aware TraversableResolver
2021-08-11 15:55:07.160 DEBUG 10 --- [ main] o.h.v.i.e.resolver.TraversableResolvers : Instantiated JPA aware TraversableResolver of type org.hibernate.validator.internal.engine.resolver.JPATraversableResolver.
2021-08-11 15:55:07.162 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : Trying to load META-INF/validation.xml for XML based Validator configuration.
2021-08-11 15:55:07.162 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via TCCL
2021-08-11 15:55:07.163 DEBUG 10 --- [ main] o.h.v.i.xml.config.ResourceLoaderHelper : Trying to load META-INF/validation.xml via Hibernate Validator's class loader
2021-08-11 15:55:07.163 DEBUG 10 --- [ main] o.h.v.i.xml.config.ValidationXmlParser : No META-INF/validation.xml found. Using annotation based configuration only.
2021-08-11 15:55:07.178 DEBUG 10 --- [ main] .h.v.m.ResourceBundleMessageInterpolator : Loaded expression factory via original TCCL
2021-08-11 15:55:07.178 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.messageinterpolation.ResourceBundleMessageInterpolator as ValidatorFactory-scoped message interpolator.
2021-08-11 15:55:07.178 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.resolver.JPATraversableResolver as ValidatorFactory-scoped traversable resolver.
2021-08-11 15:55:07.178 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.util.ExecutableParameterNameProvider as ValidatorFactory-scoped parameter name provider.
2021-08-11 15:55:07.178 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.DefaultClockProvider as ValidatorFactory-scoped clock provider.
2021-08-11 15:55:07.178 DEBUG 10 --- [ main] o.h.v.i.engine.ValidatorFactoryImpl : HV000234: Using org.hibernate.validator.internal.engine.scripting.DefaultScriptEvaluatorFactory as ValidatorFactory-scoped script evaluator factory.
2021-08-11 15:55:07.331 WARN 10 --- [ main] org.glassfish.jersey.internal.Errors : The following warnings have been detected: WARNING: The (sub)resource method getC2Qgremlin in org.onap.aai.rest.CQ2Gremlin contains empty path annotation.
WARNING: The (sub)resource method getC2Qgremlin in org.onap.aai.rest.CQ2GremlinTest contains empty path annotation.
WARNING: A HTTP GET method, public javax.ws.rs.core.Response org.onap.aai.rest.RecentAPIConsumer.getRecentData(java.lang.String,java.lang.String,java.lang.String,javax.ws.rs.core.HttpHeaders,javax.servlet.http.HttpServletRequest,javax.ws.rs.core.UriInfo), should not consume any entity.
2021-08-11 15:55:07.340 DEBUG 10 --- [ main] org.onap.aai.config.ErrorHandler : Filter 'errorHandler' configured for use
2021-08-11 15:55:07.848 INFO 10 --- [ main] org.onap.aai.TraversalApp : Application 'aai-traversal' is running on 8446!
2021-08-11 15:55:07.848 DEBUG 10 --- [ main] org.onap.aai.TraversalApp : Traversal MicroService Started
Traversal Microservice Started
2021-08-11 15:55:31.976 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:55:31.980 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:55:38.150 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:55:38.152 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:56:01.981 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:56:01.984 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:56:03.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:56:03.187 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:56:31.986 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:56:31.989 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:56:48.313 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:56:48.316 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:57:01.990 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:57:01.993 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:57:28.378 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:57:28.381 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:57:31.999 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:57:32.003 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:57:51.206 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE dox, scheduling delivery
2021-08-11 15:57:52.228 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox
2021-08-11 15:57:52.276 DEBUG 10 --- [luster-worker-2] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:57:52.284 DEBUG 10 --- [luster-worker-2] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 8 milliseconds
2021-08-11 15:57:52.396 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE zusammen_dox, scheduling delivery
2021-08-11 15:57:52.772 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE workflow, scheduling delivery
2021-08-11 15:57:53.694 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED KEYSPACE zusammen_workflow, scheduling delivery
2021-08-11 15:57:54.696 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for everything.null (null)
2021-08-11 15:57:54.901 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:57:54.908 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 7 milliseconds
2021-08-11 15:57:55.163 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TYPE dox.version, scheduling delivery
2021-08-11 15:57:56.164 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.version (TYPE)
2021-08-11 15:57:57.560 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE workflow.unique_value, scheduling delivery
2021-08-11 15:57:58.176 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TYPE dox.user_candidate_version, scheduling delivery
2021-08-11 15:57:59.094 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.item, scheduling delivery
2021-08-11 15:57:59.388 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.version_info, scheduling delivery
2021-08-11 15:57:59.984 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.version, scheduling delivery
2021-08-11 15:58:00.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.version_info_deleted, scheduling delivery
2021-08-11 15:58:01.177 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.version_elements, scheduling delivery
2021-08-11 15:58:02.178 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for everything.null (null)
2021-08-11 15:58:02.228 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:58:02.237 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 8 milliseconds
2021-08-11 15:58:02.542 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.unique_value, scheduling delivery
2021-08-11 15:58:03.462 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:58:03.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:58:03.543 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.unique_value (TABLE)
2021-08-11 15:58:03.586 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element_namespace, scheduling delivery
2021-08-11 15:58:03.977 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.package_details, scheduling delivery
2021-08-11 15:58:04.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.vsp_enriched_service_template, scheduling delivery
2021-08-11 15:58:04.997 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element, scheduling delivery
2021-08-11 15:58:05.772 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element_synchronization_state, scheduling delivery
2021-08-11 15:58:06.677 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.element_stage, scheduling delivery
2021-08-11 15:58:07.678 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for everything.null (null)
2021-08-11 15:58:07.715 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:58:07.722 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 7 milliseconds
2021-08-11 15:58:08.469 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:58:08.471 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:58:08.494 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.vsp_enriched_service_artifact, scheduling delivery
2021-08-11 15:58:08.790 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_workflow.version_stage, scheduling delivery
2021-08-11 15:58:09.788 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.application_config, scheduling delivery
2021-08-11 15:58:10.788 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for everything.null (null)
2021-08-11 15:58:10.826 DEBUG 10 --- [luster-worker-2] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:58:10.833 DEBUG 10 --- [luster-worker-2] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 7 milliseconds
2021-08-11 15:58:10.969 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.action, scheduling delivery
2021-08-11 15:58:11.720 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2021-08-11 15:58:12.721 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2021-08-11 15:58:12.896 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2021-08-11 15:58:13.679 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2021-08-11 15:58:14.680 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2021-08-11 15:58:15.621 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2021-08-11 15:58:16.622 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2021-08-11 15:58:16.817 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2021-08-11 15:58:17.817 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2021-08-11 15:58:18.606 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.ecompcomponent, scheduling delivery
2021-08-11 15:58:19.606 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.ecompcomponent (TABLE)
2021-08-11 15:58:20.527 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT UPDATED TABLE dox.action, scheduling delivery
2021-08-11 15:58:21.527 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action (TABLE)
2021-08-11 15:58:21.982 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.action_artifact, scheduling delivery
2021-08-11 15:58:22.983 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.action_artifact (TABLE)
2021-08-11 15:58:23.723 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.activity_log, scheduling delivery
2021-08-11 15:58:24.724 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.activity_log (TABLE)
2021-08-11 15:58:25.498 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.healing, scheduling delivery
2021-08-11 15:58:26.499 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.healing (TABLE)
2021-08-11 15:58:26.671 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.migration, scheduling delivery
2021-08-11 15:58:27.672 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.migration (TABLE)
2021-08-11 15:58:27.892 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.item_permissions, scheduling delivery
2021-08-11 15:58:28.893 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.item_permissions (TABLE)
2021-08-11 15:58:29.105 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.user_permission_items, scheduling delivery
2021-08-11 15:58:30.106 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.user_permission_items (TABLE)
2021-08-11 15:58:30.970 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.notification_subscribers, scheduling delivery
2021-08-11 15:58:31.924 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.last_notification, scheduling delivery
2021-08-11 15:58:32.925 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox
2021-08-11 15:58:32.940 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:58:32.950 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 10 milliseconds
2021-08-11 15:58:33.479 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.notifications, scheduling delivery
2021-08-11 15:58:34.479 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for dox.notifications (TABLE)
2021-08-11 15:58:34.827 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE dox.vsp_merge_hint, scheduling delivery
2021-08-11 15:58:35.777 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.item, scheduling delivery
2021-08-11 15:58:36.599 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.version, scheduling delivery
2021-08-11 15:58:37.600 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for everything.null (null)
2021-08-11 15:58:37.628 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:58:37.635 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 6 milliseconds
2021-08-11 15:58:37.735 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.version_elements, scheduling delivery
2021-08-11 15:58:38.735 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.version_elements (TABLE)
2021-08-11 15:58:38.871 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element_namespace, scheduling delivery
2021-08-11 15:58:39.662 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element, scheduling delivery
2021-08-11 15:58:40.663 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox
2021-08-11 15:58:40.670 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:58:40.677 DEBUG 10 --- [luster-worker-3] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 7 milliseconds
2021-08-11 15:58:40.916 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element_synchronization_state, scheduling delivery
2021-08-11 15:58:41.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.element_stage, scheduling delivery
2021-08-11 15:58:42.688 DEBUG 10 --- [luster-worker-2] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox
2021-08-11 15:58:42.694 DEBUG 10 --- [luster-worker-2] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 15:58:42.699 DEBUG 10 --- [luster-worker-2] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 5 milliseconds
2021-08-11 15:58:42.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT CREATED TABLE zusammen_dox.version_stage, scheduling delivery
2021-08-11 15:58:43.711 DEBUG 10 --- [luster-worker-3] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema for zusammen_dox.version_stage (TABLE)
2021-08-11 15:58:53.594 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:58:53.596 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:59:13.715 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:59:13.718 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 15:59:43.721 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 15:59:43.723 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:00:13.726 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:00:13.728 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:00:13.761 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:00:13.762 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:00:43.737 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:00:43.739 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:00:43.765 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:00:43.767 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:01:13.757 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:01:13.760 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:01:13.769 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:01:13.771 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:01:18.835 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:01:18.837 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:01:38.889 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:01:38.892 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:01:43.763 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:01:43.765 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:02:03.965 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:02:03.968 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:02:13.768 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:02:13.770 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:02:43.775 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:02:43.778 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:02:49.003 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:02:49.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:02:54.014 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:02:54.016 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:03:13.779 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:03:13.781 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:03:19.005 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:03:19.007 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:03:43.783 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:03:43.785 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:03:49.175 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:03:49.177 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:04:13.794 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:04:13.797 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:04:34.300 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:04:34.302 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:04:43.800 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:04:43.802 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:04:44.297 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:04:44.299 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:05:13.804 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:05:13.890 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:05:34.394 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:05:34.396 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:05:43.891 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:05:43.893 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:05:49.414 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:05:49.416 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:06:13.898 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:06:13.902 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:06:29.462 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:06:29.464 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:06:43.903 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:06:43.905 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:07:04.510 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:07:04.511 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:07:13.906 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:07:13.908 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:07:43.917 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:07:43.920 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:07:59.655 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:07:59.657 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:08:13.925 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:08:13.927 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:08:14.669 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:08:14.671 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:08:29.683 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:08:29.686 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:08:43.934 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:08:43.936 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:09:13.942 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:09:13.945 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:09:34.874 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:09:34.877 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:09:43.945 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:09:43.947 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:09:49.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:09:49.945 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:10:13.953 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:10:13.956 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:10:29.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:10:29.987 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:10:43.961 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:10:43.963 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:11:10.014 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:11:10.016 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:11:13.966 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:11:13.968 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:11:43.969 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:11:43.971 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:12:13.978 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:12:13.979 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:12:30.189 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:12:30.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:12:43.981 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:12:43.983 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:12:45.203 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:12:45.205 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:12:59.262 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:12:59.263 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:12:59.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 16:12:59.266 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-1, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:12:59.268 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-1, inFlight=0, closed=true] closing connection
2021-08-11 16:12:59.268 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 16:12:59.273 DEBUG 10 --- [luster-worker-4] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:12:59.274 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 16:12:59.367 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 16:12:59.367 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 16:12:59.973 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.110.137:9042, scheduling delivery
2021-08-11 16:12:59.973 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event DOWN
2021-08-11 16:13:00.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:13:00.063 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:13:00.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:13:00.063 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:13:00.064 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 16:13:00.064 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:13:00.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:00.064 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-2, inFlight=0, closed=true] closing connection
2021-08-11 16:13:00.064 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 16:13:00.064 DEBUG 10 --- [luster-worker-5] c.d.driver.core.ControlConnection : [Control connection] cassandra/10.242.110.137:9042 is down/removed and it was the control host, triggering reconnect
2021-08-11 16:13:00.064 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-1, inFlight=0, closed=true] closing connection
2021-08-11 16:13:00.065 DEBUG 10 --- [er-nio-worker-0] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 0ms
2021-08-11 16:13:00.065 DEBUG 10 --- [luster-worker-5] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 0ms
2021-08-11 16:13:00.065 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Host.STATES : [Control connection] starting reconnection attempt
2021-08-11 16:13:00.066 DEBUG 10 --- [er-nio-worker-0] c.d.d.core.AbstractReconnectionHandler : Found another already active handler, cancelling
2021-08-11 16:13:00.066 DEBUG 10 --- [luster-worker-5] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:13:00.066 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 16:13:00.066 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:13:00.066 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Got cancelled, stopping
2021-08-11 16:13:00.071 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-2, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:13:00.078 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-2, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:00.078 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:13:00.078 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-2, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:13:00.078 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-2, inFlight=0, closed=true] closing connection
2021-08-11 16:13:00.079 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-2, inFlight=0, closed=true] has already terminated
2021-08-11 16:13:00.079 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:00.079 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 1000 ms
2021-08-11 16:13:00.274 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:00.277 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-2, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:13:00.278 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-2, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:13:00.278 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-2, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:00.278 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-2, inFlight=0, closed=true] closing connection
2021-08-11 16:13:00.278 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:13:00.278 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:13:00.278 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 16:13:00.351 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [677387623-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2faff436 at consistency null
2021-08-11 16:13:00.351 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [677387623-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:00.449 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1887613598-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21be1346 at consistency null
2021-08-11 16:13:00.449 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1887613598-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:00.552 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [179081090-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75b01c29 at consistency null
2021-08-11 16:13:00.552 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [179081090-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:00.747 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1560573155-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@696a10c2 at consistency null
2021-08-11 16:13:00.747 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1560573155-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:00.953 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1934465559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39b4479d at consistency null
2021-08-11 16:13:00.953 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1934465559-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:00.974 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:13:00.974 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:13:01.066 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:01.070 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-3, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:13:01.070 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-3, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:13:01.070 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-3, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:01.070 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:13:01.070 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-3, inFlight=0, closed=true] closing connection
2021-08-11 16:13:01.071 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:13:01.071 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 16:13:01.080 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:13:01.081 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-3, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:13:01.152 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-3, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:01.153 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:13:01.153 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-3, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:13:01.153 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-3, inFlight=0, closed=true] closing connection
2021-08-11 16:13:01.153 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-3, inFlight=0, closed=true] has already terminated
2021-08-11 16:13:01.153 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:01.154 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 2000 ms
2021-08-11 16:13:01.156 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [381604002-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@684147ce at consistency null
2021-08-11 16:13:01.156 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [381604002-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:01.360 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2013306216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d655fbb at consistency null
2021-08-11 16:13:01.360 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2013306216-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:01.764 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [106782183-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2430a0d5 at consistency null
2021-08-11 16:13:01.764 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [106782183-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:02.168 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [425718297-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3f238361 at consistency null
2021-08-11 16:13:02.168 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [425718297-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:02.279 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:02.282 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-3, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:13:02.282 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-3, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:13:02.282 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-3, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:02.282 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-3, inFlight=0, closed=true] closing connection
2021-08-11 16:13:02.282 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:13:02.282 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:13:02.282 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 16:13:02.972 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2038696528-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5dbedc7c at consistency null
2021-08-11 16:13:02.972 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2038696528-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:02.975 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:12:54.762Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:13:03.071 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:03.073 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-4, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:13:03.074 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-4, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:13:03.074 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-4, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:03.074 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-4, inFlight=0, closed=true] closing connection
2021-08-11 16:13:03.074 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:13:03.074 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:13:03.074 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 16:13:03.154 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:13:03.156 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-4, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:13:03.159 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-4, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:03.159 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:13:03.159 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-4, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:13:03.159 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-4, inFlight=0, closed=true] closing connection
2021-08-11 16:13:03.160 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-4, inFlight=0, closed=true] has already terminated
2021-08-11 16:13:03.160 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:03.160 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 4000 ms
2021-08-11 16:13:06.283 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:06.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-4, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:13:06.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-4, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:13:06.286 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-4, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:06.287 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-4, inFlight=0, closed=true] closing connection
2021-08-11 16:13:06.287 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:13:06.287 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:13:06.287 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 16:13:07.075 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:07.077 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-5, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:13:07.077 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-5, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:13:07.077 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-5, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:07.077 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-5, inFlight=0, closed=true] closing connection
2021-08-11 16:13:07.077 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:13:07.077 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:13:07.077 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 16:13:07.160 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:13:07.162 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-5, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:13:07.165 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-5, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:07.165 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:13:07.165 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-5, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:13:07.165 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-5, inFlight=0, closed=true] closing connection
2021-08-11 16:13:07.166 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-5, inFlight=0, closed=true] has already terminated
2021-08-11 16:13:07.166 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:07.166 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 8000 ms
2021-08-11 16:13:07.979 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [85140437-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5bf8f1e3 at consistency null
2021-08-11 16:13:07.979 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [85140437-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.007 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1775101173-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59e829af at consistency null
2021-08-11 16:13:08.007 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1775101173-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.035 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [266603759-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43f40116 at consistency null
2021-08-11 16:13:08.035 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [266603759-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1458262473-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34379ae9 at consistency null
2021-08-11 16:13:08.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1458262473-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.091 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1257950538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c8a6891 at consistency null
2021-08-11 16:13:08.091 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1257950538-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.119 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1284928172-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78ebb7c at consistency null
2021-08-11 16:13:08.119 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1284928172-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.172 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1664781477-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fe699b6 at consistency null
2021-08-11 16:13:08.172 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1664781477-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.276 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1876163602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7057b63b at consistency null
2021-08-11 16:13:08.276 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1876163602-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.379 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1935666239-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16a1308e at consistency null
2021-08-11 16:13:08.379 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1935666239-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.482 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [212067306-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64063c7d at consistency null
2021-08-11 16:13:08.482 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [212067306-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:08.685 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [994777471-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f162262 at consistency null
2021-08-11 16:13:08.685 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [994777471-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:09.088 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1258836328-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@123db1d5 at consistency null
2021-08-11 16:13:09.089 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1258836328-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:09.492 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1195008546-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d07d65c at consistency null
2021-08-11 16:13:09.492 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1195008546-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:09.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1419081116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5837e655 at consistency null
2021-08-11 16:13:09.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1419081116-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:10.700 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [938872464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@365cca03 at consistency null
2021-08-11 16:13:10.700 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [938872464-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:11.503 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1056741911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49107604 at consistency null
2021-08-11 16:13:11.504 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1056741911-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:11.505 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:12:54.762Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:13:14.287 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:14.290 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-5, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:13:14.290 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-5, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:13:14.290 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-5, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:14.290 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-5, inFlight=0, closed=true] closing connection
2021-08-11 16:13:14.290 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:13:14.290 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:13:14.291 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 16:13:15.078 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:15.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-6, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:13:15.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-6, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:13:15.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-6, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:15.080 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-6, inFlight=0, closed=true] closing connection
2021-08-11 16:13:15.080 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:13:15.081 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:13:15.081 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 16:13:15.166 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:13:15.168 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-6, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:13:15.171 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-6, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:15.171 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:13:15.171 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-6, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:13:15.171 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-6, inFlight=0, closed=true] closing connection
2021-08-11 16:13:15.171 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-6, inFlight=0, closed=true] has already terminated
2021-08-11 16:13:15.171 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:15.171 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 16000 ms
2021-08-11 16:13:16.509 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [717477016-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a96fc4c at consistency null
2021-08-11 16:13:16.510 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [717477016-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:16.563 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1434807753-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1df4b805 at consistency null
2021-08-11 16:13:16.563 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1434807753-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:16.666 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1384151911-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4da8656a at consistency null
2021-08-11 16:13:16.666 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1384151911-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:16.869 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [150372098-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c29a8d at consistency null
2021-08-11 16:13:16.870 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [150372098-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:17.273 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1009061264-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a7b62a5 at consistency null
2021-08-11 16:13:17.273 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1009061264-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:17.676 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1110496504-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@72f7cf7f at consistency null
2021-08-11 16:13:17.676 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1110496504-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:18.080 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [322618522-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c7a677e at consistency null
2021-08-11 16:13:18.080 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [322618522-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:18.884 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [680503184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fff9c8e at consistency null
2021-08-11 16:13:18.884 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [680503184-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:19.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1012037276-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e00b59e at consistency null
2021-08-11 16:13:19.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1012037276-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:19.688 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:12:54.762Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:13:24.694 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2013393658-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32c01116 at consistency null
2021-08-11 16:13:24.695 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2013393658-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:24.749 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1037756276-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1daed1a7 at consistency null
2021-08-11 16:13:24.749 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1037756276-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:24.853 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1757181048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e263570 at consistency null
2021-08-11 16:13:24.853 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1757181048-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:25.056 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1207245157-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7571dd6f at consistency null
2021-08-11 16:13:25.056 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1207245157-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:25.460 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [815048003-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@478b2a76 at consistency null
2021-08-11 16:13:25.460 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [815048003-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:26.264 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1520375138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c43e8c9 at consistency null
2021-08-11 16:13:26.264 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1520375138-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:27.067 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1368110589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13f1c584 at consistency null
2021-08-11 16:13:27.067 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1368110589-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:28.670 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1046130840-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6810e830 at consistency null
2021-08-11 16:13:28.671 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1046130840-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:28.672 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:12:54.762Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:13:30.291 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:30.294 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-6, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:13:30.295 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-6, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:13:30.295 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-6, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:30.295 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-6, inFlight=0, closed=true] closing connection
2021-08-11 16:13:30.295 DEBUG 10 --- [luster-worker-4] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:13:30.295 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:13:30.295 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 16:13:31.081 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:13:31.083 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-7, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:13:31.084 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-7, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:13:31.084 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-7, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:13:31.084 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-7, inFlight=0, closed=true] closing connection
2021-08-11 16:13:31.084 DEBUG 10 --- [luster-worker-5] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:13:31.085 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:13:31.085 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 16:13:31.172 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:13:31.174 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-7, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:13:31.179 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-7, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:31.179 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:13:31.179 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-7, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:13:31.179 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-7, inFlight=0, closed=true] closing connection
2021-08-11 16:13:31.179 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-7, inFlight=0, closed=true] has already terminated
2021-08-11 16:13:31.179 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:13:31.180 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 32000 ms
2021-08-11 16:13:33.677 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [737256621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58a03ca0 at consistency null
2021-08-11 16:13:33.677 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [737256621-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:33.705 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [157778100-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@379e0eec at consistency null
2021-08-11 16:13:33.706 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [157778100-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:33.734 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [89126968-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74b52786 at consistency null
2021-08-11 16:13:33.734 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [89126968-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:33.762 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1861899897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31f34e19 at consistency null
2021-08-11 16:13:33.762 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1861899897-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:33.816 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1208501191-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68ef31bb at consistency null
2021-08-11 16:13:33.816 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1208501191-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:33.869 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2103380050-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fc1bb24 at consistency null
2021-08-11 16:13:33.869 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2103380050-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:33.971 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [128692729-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7295304d at consistency null
2021-08-11 16:13:33.972 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [128692729-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:34.175 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [996042475-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59bd3b80 at consistency null
2021-08-11 16:13:34.175 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [996042475-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:34.378 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [826416188-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@48ceabe5 at consistency null
2021-08-11 16:13:34.379 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [826416188-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:34.784 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [886738910-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a743143 at consistency null
2021-08-11 16:13:34.784 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [886738910-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:35.588 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1856296133-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31b87318 at consistency null
2021-08-11 16:13:35.588 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1856296133-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:36.392 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [646287044-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e93f94 at consistency null
2021-08-11 16:13:36.392 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [646287044-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:37.195 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [126342612-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3cc53dcf at consistency null
2021-08-11 16:13:37.195 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [126342612-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:37.197 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:12:54.762Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:13:42.202 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1740840581-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77f0ed3d at consistency null
2021-08-11 16:13:42.203 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1740840581-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:42.256 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [797568068-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a26068 at consistency null
2021-08-11 16:13:42.256 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [797568068-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:42.310 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [628143327-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d7ffad7 at consistency null
2021-08-11 16:13:42.310 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [628143327-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:42.414 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [484502736-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58419df2 at consistency null
2021-08-11 16:13:42.414 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [484502736-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:42.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1672019711-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28a80ac2 at consistency null
2021-08-11 16:13:42.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1672019711-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:42.821 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [52675842-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e9f8e9f at consistency null
2021-08-11 16:13:42.821 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [52675842-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:43.224 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [592793589-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c8472d9 at consistency null
2021-08-11 16:13:43.225 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [592793589-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:13:44.028 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [944283939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@587ed0b6 at consistency null
2021-08-11 16:13:44.029 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [944283939-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:14:02.296 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:14:02.299 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:14:03.085 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:14:03.087 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:14:06.464 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:14:06.465 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Successful reconnection to cassandra/10.242.110.137:9042, setting host UP
2021-08-11 16:14:06.465 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 16:14:06.465 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:14:06.466 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 16:14:06.968 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:14:06.969 DEBUG 10 --- [-reconnection-0] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:14:06.972 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 16:14:06.974 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:14:06.974 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:14:06.976 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:14:08.565 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:14:08.566 DEBUG 10 --- [-reconnection-0] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2021-08-11 16:14:08.576 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:14:08.581 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 5 milliseconds
2021-08-11 16:14:08.587 DEBUG 10 --- [-reconnection-0] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2021-08-11 16:14:08.649 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:14:08.649 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Successful reconnection to /10.242.184.131:9042, setting host UP
2021-08-11 16:14:08.649 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 16:14:08.649 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:14:08.649 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 16:14:08.801 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:14:08.806 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 4 milliseconds
2021-08-11 16:14:08.812 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] established to cassandra/10.242.110.137:9042
2021-08-11 16:14:08.812 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:14:09.149 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:14:09.150 DEBUG 10 --- [-reconnection-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:14:09.150 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 16:14:09.150 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:14:38.787 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:14:38.790 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:14:50.885 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:14:50.887 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:15:05.984 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:15:05.986 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:15:08.813 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:15:08.816 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:15:36.073 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:15:36.076 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:15:38.841 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:15:38.843 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:16:08.872 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:16:08.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:16:38.902 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:16:38.904 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:17:08.932 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:17:08.934 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:17:16.384 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:17:16.386 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:17:38.956 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:17:38.961 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:17:41.469 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:17:41.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=1, closed=false] heartbeat query succeeded
2021-08-11 16:17:51.492 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:17:51.546 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:18:08.987 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:18:08.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:18:26.577 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:18:26.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:18:39.010 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:18:39.012 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:19:09.037 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:19:09.039 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:19:26.769 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:19:26.772 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:19:31.850 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:19:31.853 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:19:39.065 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:19:39.067 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:01.970 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:20:01.972 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:09.094 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:20:09.096 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:16.982 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:20:16.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:31.986 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:20:31.988 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:39.125 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:20:39.127 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:47.013 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:20:47.015 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:57.021 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:20:57.062 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:20:59.261 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:20:59.261 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:20:59.261 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 16:20:59.261 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-7, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:20:59.262 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-7, inFlight=0, closed=true] closing connection
2021-08-11 16:20:59.263 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 16:20:59.263 DEBUG 10 --- [luster-worker-6] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:20:59.264 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 16:20:59.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 16:20:59.369 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 16:20:59.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.110.137:9042, scheduling delivery
2021-08-11 16:20:59.867 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event DOWN
2021-08-11 16:20:59.872 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:20:59.873 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:20:59.873 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 16:20:59.873 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:20:59.873 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-8, inFlight=0, closed=true] closing connection
2021-08-11 16:20:59.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:20:59.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:20:59.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:20:59.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-9, inFlight=0, closed=true] closing connection
2021-08-11 16:20:59.875 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 16:20:59.875 DEBUG 10 --- [luster-worker-7] c.d.driver.core.ControlConnection : [Control connection] cassandra/10.242.110.137:9042 is down/removed and it was the control host, triggering reconnect
2021-08-11 16:20:59.875 DEBUG 10 --- [luster-worker-7] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 0ms
2021-08-11 16:20:59.875 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Host.STATES : [Control connection] starting reconnection attempt
2021-08-11 16:20:59.875 DEBUG 10 --- [luster-worker-7] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:20:59.875 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 16:20:59.875 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:20:59.877 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-8, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:20:59.879 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-8, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:20:59.879 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:20:59.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-8, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:20:59.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-8, inFlight=0, closed=true] closing connection
2021-08-11 16:20:59.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-8, inFlight=0, closed=true] has already terminated
2021-08-11 16:20:59.880 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:20:59.880 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 1000 ms
2021-08-11 16:21:00.264 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:00.267 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-8, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:21:00.267 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-8, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:21:00.267 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-8, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:00.267 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-8, inFlight=0, closed=true] closing connection
2021-08-11 16:21:00.267 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:21:00.268 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:21:00.268 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 16:21:00.867 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:21:00.867 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:21:00.875 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:00.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-10, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:21:00.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-10, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:21:00.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-10, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:00.877 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-10, inFlight=0, closed=true] closing connection
2021-08-11 16:21:00.877 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:21:00.878 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:21:00.878 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 16:21:00.881 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:21:00.882 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-9, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:21:00.883 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-9, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:00.883 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:21:00.883 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-9, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:21:00.884 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-9, inFlight=0, closed=true] closing connection
2021-08-11 16:21:00.884 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-9, inFlight=0, closed=true] has already terminated
2021-08-11 16:21:00.884 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:00.884 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 2000 ms
2021-08-11 16:21:02.051 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [219132561-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ef07041 at consistency null
2021-08-11 16:21:02.051 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [219132561-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:02.079 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [472590742-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46466fed at consistency null
2021-08-11 16:21:02.079 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [472590742-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:02.132 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [881968922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ff7a50b at consistency null
2021-08-11 16:21:02.132 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [881968922-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:02.185 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1788569746-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38946f54 at consistency null
2021-08-11 16:21:02.185 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1788569746-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:02.268 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:02.270 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-9, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:21:02.271 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-9, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:21:02.271 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-9, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:02.271 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-9, inFlight=0, closed=true] closing connection
2021-08-11 16:21:02.271 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:21:02.272 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:21:02.272 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 16:21:02.288 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [451986538-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12aa70da at consistency null
2021-08-11 16:21:02.288 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [451986538-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:02.491 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1599220027-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@95f5314 at consistency null
2021-08-11 16:21:02.491 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1599220027-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:02.878 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:02.880 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-11, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:21:02.880 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-11, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:21:02.881 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-11, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:02.881 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-11, inFlight=0, closed=true] closing connection
2021-08-11 16:21:02.881 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:21:02.881 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:21:02.881 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 16:21:02.884 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:21:02.885 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-10, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:21:02.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-10, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:02.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:21:02.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-10, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:21:02.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-10, inFlight=0, closed=true] closing connection
2021-08-11 16:21:02.888 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-10, inFlight=0, closed=true] has already terminated
2021-08-11 16:21:02.888 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:02.888 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 4000 ms
2021-08-11 16:21:02.894 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1698798812-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@174950ca at consistency null
2021-08-11 16:21:02.894 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1698798812-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:03.698 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1412409118-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1db540e1 at consistency null
2021-08-11 16:21:03.698 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1412409118-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:04.501 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1222699621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5cd8df2d at consistency null
2021-08-11 16:21:04.501 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1222699621-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:05.305 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [104139187-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fecf051 at consistency null
2021-08-11 16:21:05.305 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [104139187-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:05.307 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:20:56.538Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:21:06.272 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:06.274 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-10, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:21:06.274 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-10, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:21:06.275 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-10, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:06.275 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-10, inFlight=0, closed=true] closing connection
2021-08-11 16:21:06.275 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:21:06.275 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:21:06.275 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 16:21:06.881 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:06.884 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-12, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:21:06.884 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-12, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:21:06.884 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-12, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:06.884 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-12, inFlight=0, closed=true] closing connection
2021-08-11 16:21:06.884 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:21:06.885 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:21:06.885 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 16:21:06.889 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:21:06.890 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-11, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:21:06.892 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-11, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:06.892 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:21:06.892 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-11, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:21:06.892 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-11, inFlight=0, closed=true] closing connection
2021-08-11 16:21:06.893 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-11, inFlight=0, closed=true] has already terminated
2021-08-11 16:21:06.893 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:06.893 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 8000 ms
2021-08-11 16:21:10.312 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1870381781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f4050a5 at consistency null
2021-08-11 16:21:10.312 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1870381781-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.340 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1178963827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17665d64 at consistency null
2021-08-11 16:21:10.340 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1178963827-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.369 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [163182108-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@144868e4 at consistency null
2021-08-11 16:21:10.369 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [163182108-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [669982217-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fdcaba3 at consistency null
2021-08-11 16:21:10.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [669982217-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.425 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [372448587-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@52c40785 at consistency null
2021-08-11 16:21:10.425 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [372448587-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2087134953-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77ecfdc4 at consistency null
2021-08-11 16:21:10.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2087134953-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.506 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [984098415-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2034a495 at consistency null
2021-08-11 16:21:10.506 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [984098415-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.609 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1709729893-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4adf9283 at consistency null
2021-08-11 16:21:10.609 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1709729893-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:10.811 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1799354582-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38554c72 at consistency null
2021-08-11 16:21:10.812 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1799354582-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:11.014 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [782945078-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28c79461 at consistency null
2021-08-11 16:21:11.014 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [782945078-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:11.417 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1430920146-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d529de0 at consistency null
2021-08-11 16:21:11.417 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1430920146-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:12.220 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1942481502-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1baf5943 at consistency null
2021-08-11 16:21:12.220 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1942481502-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:13.023 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1739212216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@90ba098 at consistency null
2021-08-11 16:21:13.024 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1739212216-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:13.025 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:20:56.538Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:21:14.276 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:14.278 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-11, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:21:14.278 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-11, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:21:14.279 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-11, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:14.279 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-11, inFlight=0, closed=true] closing connection
2021-08-11 16:21:14.279 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:21:14.279 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:21:14.279 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 16:21:14.885 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:14.887 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-13, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:21:14.887 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-13, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:21:14.887 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-13, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:14.887 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-13, inFlight=0, closed=true] closing connection
2021-08-11 16:21:14.888 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:21:14.888 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 16:21:14.888 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:21:14.893 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:21:14.894 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-12, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:21:14.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-12, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:14.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:21:14.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-12, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:21:14.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-12, inFlight=0, closed=true] closing connection
2021-08-11 16:21:14.898 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-12, inFlight=0, closed=true] has already terminated
2021-08-11 16:21:14.898 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:14.898 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 16000 ms
2021-08-11 16:21:18.029 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [530694400-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@457b4a1b at consistency null
2021-08-11 16:21:18.029 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [530694400-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:18.082 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [447794605-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11203d4c at consistency null
2021-08-11 16:21:18.082 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [447794605-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:18.186 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1458127295-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56566755 at consistency null
2021-08-11 16:21:18.186 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1458127295-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:18.389 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1666931039-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9228c43 at consistency null
2021-08-11 16:21:18.389 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1666931039-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:18.592 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1919102453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@161634ee at consistency null
2021-08-11 16:21:18.593 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1919102453-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:18.996 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1847745457-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64ca13d at consistency null
2021-08-11 16:21:18.996 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1847745457-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:19.799 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1918507891-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6345126e at consistency null
2021-08-11 16:21:19.799 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1918507891-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:21.403 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1274160582-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7e18dad1 at consistency null
2021-08-11 16:21:21.403 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1274160582-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:21.404 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:20:56.538Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:21:26.409 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [546678216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@590e30b8 at consistency null
2021-08-11 16:21:26.409 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [546678216-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:26.462 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1553370552-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3aa837ed at consistency null
2021-08-11 16:21:26.462 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1553370552-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:26.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2125829338-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71b594b4 at consistency null
2021-08-11 16:21:26.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2125829338-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:26.650 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [615731935-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49430601 at consistency null
2021-08-11 16:21:26.650 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [615731935-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:26.747 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [696588388-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47c1072e at consistency null
2021-08-11 16:21:26.747 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [696588388-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:26.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1092004769-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@16e372d9 at consistency null
2021-08-11 16:21:26.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1092004769-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:26.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [332919837-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74c24f18 at consistency null
2021-08-11 16:21:26.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [332919837-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:27.153 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [768370962-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fb28227 at consistency null
2021-08-11 16:21:27.154 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [768370962-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:27.557 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [747307767-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40051c88 at consistency null
2021-08-11 16:21:27.557 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [747307767-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:27.960 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1479868260-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d7dcbeb at consistency null
2021-08-11 16:21:27.960 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1479868260-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:28.764 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1953472355-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ba1e7d3 at consistency null
2021-08-11 16:21:28.764 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1953472355-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:29.567 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [975563781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3019eb3a at consistency null
2021-08-11 16:21:29.567 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [975563781-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:30.280 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:30.282 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-12, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:21:30.283 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-12, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:21:30.283 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-12, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:30.283 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-12, inFlight=0, closed=true] closing connection
2021-08-11 16:21:30.283 DEBUG 10 --- [luster-worker-6] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:21:30.284 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:21:30.284 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 16:21:30.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1313725708-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@581040ed at consistency null
2021-08-11 16:21:30.371 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1313725708-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:30.372 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:20:56.538Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:21:30.888 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:21:30.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-14, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:21:30.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-14, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:21:30.890 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-14, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:21:30.891 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-14, inFlight=0, closed=true] closing connection
2021-08-11 16:21:30.891 DEBUG 10 --- [luster-worker-7] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:21:30.891 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:21:30.891 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 16:21:30.898 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:21:30.900 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-13, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:21:30.902 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-13, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:30.902 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:21:30.902 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-13, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:21:30.902 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-13, inFlight=0, closed=true] closing connection
2021-08-11 16:21:30.903 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-13, inFlight=0, closed=true] has already terminated
2021-08-11 16:21:30.903 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:21:30.903 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 32000 ms
2021-08-11 16:21:35.377 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [830325886-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b11fa97 at consistency null
2021-08-11 16:21:35.377 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [830325886-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:35.404 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [971096501-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f2a1559 at consistency null
2021-08-11 16:21:35.404 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [971096501-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:35.432 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [741431112-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@201575bd at consistency null
2021-08-11 16:21:35.432 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [741431112-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:35.485 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [213142143-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ad85e7a at consistency null
2021-08-11 16:21:35.485 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [213142143-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:35.588 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1404379555-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ee9d6f4 at consistency null
2021-08-11 16:21:35.588 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1404379555-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:35.791 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1693363721-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e69aad at consistency null
2021-08-11 16:21:35.791 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1693363721-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:35.994 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1120087755-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a55bfb1 at consistency null
2021-08-11 16:21:35.994 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1120087755-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:36.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1821012830-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32c61b0f at consistency null
2021-08-11 16:21:36.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1821012830-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:37.200 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [382267221-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@725825d5 at consistency null
2021-08-11 16:21:37.201 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [382267221-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:38.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [130520419-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@475eeac1 at consistency null
2021-08-11 16:21:38.848 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [130520419-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:38.849 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:20:56.538Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:21:43.857 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1546208806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60517dac at consistency null
2021-08-11 16:21:43.857 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1546208806-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:43.910 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1669779070-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62cd1cca at consistency null
2021-08-11 16:21:43.910 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1669779070-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:44.013 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [845494044-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5108ebcb at consistency null
2021-08-11 16:21:44.013 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [845494044-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:44.116 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [162713790-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61b01bc9 at consistency null
2021-08-11 16:21:44.116 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [162713790-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:44.219 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [524070872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a67b64e at consistency null
2021-08-11 16:21:44.219 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [524070872-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:44.323 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1251764095-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46d084 at consistency null
2021-08-11 16:21:44.323 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1251764095-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:21:44.526 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1364878567-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@168635f4 at consistency null
2021-08-11 16:21:44.526 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1364878567-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:22:02.284 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:22:02.288 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:22:02.891 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:22:02.894 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:22:03.151 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:22:03.151 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Successful reconnection to /10.242.184.131:9042, setting host UP
2021-08-11 16:22:03.151 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 16:22:03.151 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:22:03.151 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 16:22:03.168 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:22:03.168 DEBUG 10 --- [-reconnection-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:22:03.168 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 16:22:03.168 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:22:03.168 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:22:03.170 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:22:03.852 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:22:03.853 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2021-08-11 16:22:03.864 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:22:03.869 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 5 milliseconds
2021-08-11 16:22:03.877 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2021-08-11 16:22:04.282 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:22:04.286 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 4 milliseconds
2021-08-11 16:22:04.293 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] established to /10.242.184.131:9042
2021-08-11 16:22:04.293 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:22:05.762 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:22:05.763 DEBUG 10 --- [-reconnection-0] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on cassandra/10.242.110.137:9042
2021-08-11 16:22:05.768 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Successful reconnection to cassandra/10.242.110.137:9042, setting host UP
2021-08-11 16:22:05.768 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 16:22:05.768 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:22:05.768 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 16:22:06.272 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:22:06.272 DEBUG 10 --- [-reconnection-0] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:22:06.272 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 16:22:06.272 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:22:35.770 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:22:35.772 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:22:49.884 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:22:49.886 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:23:05.781 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:23:05.784 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:23:35.812 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:23:35.814 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:23:44.997 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:23:45.000 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:24:05.823 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:24:05.826 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:24:35.833 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:24:35.835 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:24:40.080 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:24:40.082 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:25:05.857 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:25:05.860 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:25:15.121 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:25:15.123 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:25:35.881 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:25:35.883 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:26:05.907 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:26:05.910 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:26:10.221 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:26:10.228 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:26:20.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:26:20.252 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:26:35.925 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:26:35.928 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:27:05.956 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:27:05.958 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:27:10.308 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:27:10.311 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:27:35.987 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:27:35.989 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:27:45.500 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:27:45.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:28:06.014 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:28:06.016 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:28:20.588 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:28:20.590 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:28:35.677 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:28:35.679 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:28:36.043 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:28:36.045 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:28:59.159 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 16:28:59.159 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 16:28:59.249 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:28:59.250 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:28:59.250 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 16:28:59.250 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-14, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:28:59.250 DEBUG 10 --- [er-nio-worker-2] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 0ms
2021-08-11 16:28:59.250 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [Control connection] starting reconnection attempt
2021-08-11 16:28:59.250 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-14, inFlight=0, closed=true] closing connection
2021-08-11 16:28:59.250 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:28:59.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:28:59.252 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:28:59.252 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-13, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:28:59.253 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-13, inFlight=0, closed=true] closing connection
2021-08-11 16:28:59.253 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 16:28:59.253 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-16, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:28:59.253 DEBUG 10 --- [luster-worker-8] c.d.driver.core.ControlConnection : [Control connection] /10.242.184.131:9042 is down/removed and it was the control host, triggering reconnect
2021-08-11 16:28:59.254 DEBUG 10 --- [luster-worker-8] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:28:59.254 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 16:29:00.160 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:29:00.254 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:00.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-15, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:29:00.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-15, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:29:00.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-15, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:00.258 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-15, inFlight=0, closed=true] closing connection
2021-08-11 16:29:00.258 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:29:00.259 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:29:00.259 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 16:29:00.661 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:29:00.662 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:29:00.662 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 16:29:00.662 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:29:00.662 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-15, inFlight=0, closed=true] closing connection
2021-08-11 16:29:00.663 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-16, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:29:00.663 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-16, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:29:00.664 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-16, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:00.664 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-16, inFlight=0, closed=true] closing connection
2021-08-11 16:29:00.664 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 16:29:00.664 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Not terminating Connection[cassandra/10.242.110.137:9042-16, inFlight=0, closed=true]: there are still pending requests
2021-08-11 16:29:00.664 DEBUG 10 --- [luster-worker-9] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:29:00.664 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 16:29:00.665 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] error on cassandra/10.242.110.137:9042 connection, trying next host
com.datastax.driver.core.exceptions.TransportException: [cassandra/10.242.110.137:9042] Connection has been closed
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1397)
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1378)
at com.datastax.driver.core.Connection.defunct(Connection.java:570)
at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1301)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:271)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:29:00.666 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:29:00.667 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-14, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:29:00.670 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-14, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:00.670 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:29:00.670 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-14, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:29:00.670 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-14, inFlight=0, closed=true] closing connection
2021-08-11 16:29:00.670 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-14, inFlight=0, closed=true] has already terminated
2021-08-11 16:29:00.670 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:00.671 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 1000 ms
2021-08-11 16:29:00.854 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1744004340-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e72a37a at consistency null
2021-08-11 16:29:00.855 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1744004340-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:00.909 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1498250405-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76bfd4dc at consistency null
2021-08-11 16:29:00.909 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1498250405-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:01.013 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1340087715-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79c2f578 at consistency null
2021-08-11 16:29:01.013 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1340087715-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:01.217 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [727011258-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24767c33 at consistency null
2021-08-11 16:29:01.217 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [727011258-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:01.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1476221664-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f6462b4 at consistency null
2021-08-11 16:29:01.423 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1476221664-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:01.664 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:01.667 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-17, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:29:01.667 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-17, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:29:01.667 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-17, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:01.667 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:29:01.667 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-17, inFlight=0, closed=true] closing connection
2021-08-11 16:29:01.668 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:29:01.668 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 16:29:01.671 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:29:01.672 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-15, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:29:01.675 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-15, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:01.675 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:29:01.675 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-15, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:29:01.675 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-15, inFlight=0, closed=true] closing connection
2021-08-11 16:29:01.675 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-15, inFlight=0, closed=true] has already terminated
2021-08-11 16:29:01.675 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:01.676 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 2000 ms
2021-08-11 16:29:01.826 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1701293294-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e7266ea at consistency null
2021-08-11 16:29:01.826 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1701293294-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:02.259 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:02.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-16, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:29:02.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-16, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:29:02.261 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-16, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:02.262 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-16, inFlight=0, closed=true] closing connection
2021-08-11 16:29:02.262 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:29:02.262 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:29:02.262 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 16:29:02.629 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [707111140-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@527fc052 at consistency null
2021-08-11 16:29:02.629 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [707111140-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:03.668 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:03.671 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-18, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:29:03.671 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-18, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:29:03.671 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-18, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:03.671 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-18, inFlight=0, closed=true] closing connection
2021-08-11 16:29:03.671 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:29:03.672 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:29:03.672 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 16:29:03.676 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:29:03.677 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-16, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:29:03.680 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-16, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:03.680 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:29:03.680 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-16, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:29:03.680 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-16, inFlight=0, closed=true] closing connection
2021-08-11 16:29:03.681 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-16, inFlight=0, closed=true] has already terminated
2021-08-11 16:29:03.681 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:03.681 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 4000 ms
2021-08-11 16:29:04.232 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [730602117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2aaccfe2 at consistency null
2021-08-11 16:29:04.233 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [730602117-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:04.234 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:28:55.348Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:29:06.262 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:06.264 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-17, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:29:06.265 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-17, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:29:06.265 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-17, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:06.265 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-17, inFlight=0, closed=true] closing connection
2021-08-11 16:29:06.265 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:29:06.265 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:29:06.265 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 16:29:07.672 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:07.674 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-19, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:29:07.674 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-19, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:29:07.675 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-19, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:07.675 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-19, inFlight=0, closed=true] closing connection
2021-08-11 16:29:07.675 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:29:07.676 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:29:07.676 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 16:29:07.682 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:29:07.683 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-17, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:29:07.685 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-17, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:07.685 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:29:07.685 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-17, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:29:07.685 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-17, inFlight=0, closed=true] closing connection
2021-08-11 16:29:07.685 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-17, inFlight=0, closed=true] has already terminated
2021-08-11 16:29:07.686 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:07.686 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 8000 ms
2021-08-11 16:29:09.239 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [370900814-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@656238ce at consistency null
2021-08-11 16:29:09.239 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [370900814-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.291 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1313055167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5432bd0f at consistency null
2021-08-11 16:29:09.292 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1313055167-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.344 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1227244059-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40456650 at consistency null
2021-08-11 16:29:09.344 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1227244059-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [45425032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cda7265 at consistency null
2021-08-11 16:29:09.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [45425032-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.449 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1335484319-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@780a8f78 at consistency null
2021-08-11 16:29:09.450 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1335484319-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [478462346-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39918fb3 at consistency null
2021-08-11 16:29:09.503 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [478462346-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.555 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1741254299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@520d0937 at consistency null
2021-08-11 16:29:09.555 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1741254299-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.608 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1281079947-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d1b1049 at consistency null
2021-08-11 16:29:09.608 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1281079947-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1304424888-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21c62fdd at consistency null
2021-08-11 16:29:09.711 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1304424888-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:09.913 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1997402388-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@141fc041 at consistency null
2021-08-11 16:29:09.914 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1997402388-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:10.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1537008856-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e612744 at consistency null
2021-08-11 16:29:10.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1537008856-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:10.720 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1250071234-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b2df399 at consistency null
2021-08-11 16:29:10.720 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1250071234-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:11.123 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2021824539-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6b0583e6 at consistency null
2021-08-11 16:29:11.123 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2021824539-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:11.926 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1621050190-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1724b0f at consistency null
2021-08-11 16:29:11.926 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1621050190-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:11.927 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:28:55.348Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:29:14.266 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:14.268 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-18, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:29:14.269 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-18, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:29:14.269 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-18, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:14.269 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-18, inFlight=0, closed=true] closing connection
2021-08-11 16:29:14.269 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:29:14.269 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:29:14.270 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 16:29:15.677 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:15.678 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-20, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:29:15.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-20, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:29:15.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-20, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:15.679 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:29:15.679 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-20, inFlight=0, closed=true] closing connection
2021-08-11 16:29:15.679 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:29:15.680 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 16:29:15.686 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:29:15.688 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-18, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:29:15.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-18, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:15.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:29:15.690 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-18, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:29:15.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-18, inFlight=0, closed=true] closing connection
2021-08-11 16:29:15.691 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-18, inFlight=0, closed=true] has already terminated
2021-08-11 16:29:15.691 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:15.691 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 16000 ms
2021-08-11 16:29:16.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1545191841-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@75f62601 at consistency null
2021-08-11 16:29:16.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1545191841-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:16.959 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [39013474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@66cf2183 at consistency null
2021-08-11 16:29:16.959 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [39013474-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:16.987 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1853709939-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@302c9f2f at consistency null
2021-08-11 16:29:16.987 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1853709939-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:17.047 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1961031105-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e21eb3a at consistency null
2021-08-11 16:29:17.047 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1961031105-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:17.149 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1726232117-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1aa318d1 at consistency null
2021-08-11 16:29:17.150 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1726232117-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:17.353 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2127316695-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fff13f at consistency null
2021-08-11 16:29:17.353 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2127316695-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:17.756 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2051412411-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b06f3d at consistency null
2021-08-11 16:29:17.756 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2051412411-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:18.559 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [434749391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@783c005d at consistency null
2021-08-11 16:29:18.559 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [434749391-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:19.362 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1585149785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fff6891 at consistency null
2021-08-11 16:29:19.363 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1585149785-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:19.364 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:28:55.348Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:29:24.368 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1109223615-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@63c7cd32 at consistency null
2021-08-11 16:29:24.368 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1109223615-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:24.421 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [136077966-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@620f9ff0 at consistency null
2021-08-11 16:29:24.421 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [136077966-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:24.524 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1549771653-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aa95504 at consistency null
2021-08-11 16:29:24.524 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1549771653-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:24.727 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1503381224-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5eb60120 at consistency null
2021-08-11 16:29:24.727 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1503381224-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:24.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1886000628-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3c5e999 at consistency null
2021-08-11 16:29:24.931 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1886000628-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:25.334 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [453575013-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1dab060e at consistency null
2021-08-11 16:29:25.335 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [453575013-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:26.147 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [980142602-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2d9a61ca at consistency null
2021-08-11 16:29:26.147 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [980142602-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:26.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1036052265-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@434bca58 at consistency null
2021-08-11 16:29:26.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1036052265-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:27.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [489001168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6fee7a89 at consistency null
2021-08-11 16:29:27.754 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [489001168-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:27.755 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:28:55.348Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:29:30.270 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:30.272 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-19, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:29:30.273 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-19, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:29:30.273 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-19, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:30.273 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-19, inFlight=0, closed=true] closing connection
2021-08-11 16:29:30.273 DEBUG 10 --- [luster-worker-9] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:29:30.273 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:29:30.273 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 16:29:31.235 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-16, inFlight=0, closed=true] has already terminated
2021-08-11 16:29:31.680 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:29:31.682 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-21, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:29:31.682 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-21, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:29:31.682 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-21, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:29:31.682 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-21, inFlight=0, closed=true] closing connection
2021-08-11 16:29:31.682 DEBUG 10 --- [luster-worker-8] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:29:31.683 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:29:31.683 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 16:29:31.692 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:29:31.693 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-19, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:29:31.696 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-19, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:31.696 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:29:31.696 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-19, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:29:31.696 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-19, inFlight=0, closed=true] closing connection
2021-08-11 16:29:31.696 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-19, inFlight=0, closed=true] has already terminated
2021-08-11 16:29:31.696 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:29:31.697 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 32000 ms
2021-08-11 16:29:32.759 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1316272184-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49de3378 at consistency null
2021-08-11 16:29:32.759 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1316272184-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:32.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [707465318-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@557822f8 at consistency null
2021-08-11 16:29:32.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [707465318-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:32.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [283657125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b3622b1 at consistency null
2021-08-11 16:29:32.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [283657125-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:33.153 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [561705592-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ebc1054 at consistency null
2021-08-11 16:29:33.153 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [561705592-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:33.356 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1243888407-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46c1a908 at consistency null
2021-08-11 16:29:33.356 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1243888407-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:33.559 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [956213781-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@523a2cca at consistency null
2021-08-11 16:29:33.559 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [956213781-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:33.762 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1784226756-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1853bf96 at consistency null
2021-08-11 16:29:33.762 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1784226756-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:33.965 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [474408389-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f32278 at consistency null
2021-08-11 16:29:33.965 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [474408389-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:34.168 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2001836505-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d314247 at consistency null
2021-08-11 16:29:34.168 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2001836505-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:34.571 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [984754980-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bc4c203 at consistency null
2021-08-11 16:29:34.571 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [984754980-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:34.974 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [433486005-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54bbc9e1 at consistency null
2021-08-11 16:29:34.974 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [433486005-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:35.777 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [848678535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@135f8a4b at consistency null
2021-08-11 16:29:35.777 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [848678535-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:35.778 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:28:55.348Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:29:40.784 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1232148061-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e566fa4 at consistency null
2021-08-11 16:29:40.784 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1232148061-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:40.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1655828316-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43ef64b7 at consistency null
2021-08-11 16:29:40.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1655828316-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:40.947 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1216884847-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@491159ea at consistency null
2021-08-11 16:29:40.947 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1216884847-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:41.050 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1416969743-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@26f38f51 at consistency null
2021-08-11 16:29:41.050 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1416969743-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:41.254 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [368515362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b45ff64 at consistency null
2021-08-11 16:29:41.254 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [368515362-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:41.456 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2018627535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3b7ee67 at consistency null
2021-08-11 16:29:41.457 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2018627535-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:41.859 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1909691362-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@40aa39f2 at consistency null
2021-08-11 16:29:41.859 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1909691362-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:42.662 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [204168849-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7a19afb1 at consistency null
2021-08-11 16:29:42.662 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [204168849-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:43.465 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [557344831-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eb3d149 at consistency null
2021-08-11 16:29:43.465 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [557344831-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:29:43.466 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:28:55.348Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:30:02.274 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:30:02.276 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:30:03.683 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:30:03.687 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:30:04.353 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:30:04.353 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Successful reconnection to /10.242.184.131:9042, setting host UP
2021-08-11 16:30:04.353 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 16:30:04.353 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:30:04.353 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 16:30:04.448 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:30:04.449 DEBUG 10 --- [-reconnection-0] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:30:04.449 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 16:30:04.449 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:30:04.449 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:30:04.451 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:30:05.150 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:30:05.150 DEBUG 10 --- [-reconnection-0] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2021-08-11 16:30:05.164 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:30:05.169 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 5 milliseconds
2021-08-11 16:30:05.179 DEBUG 10 --- [-reconnection-0] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2021-08-11 16:30:05.518 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:30:05.653 DEBUG 10 --- [-reconnection-0] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 135 milliseconds
2021-08-11 16:30:05.661 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] established to /10.242.184.131:9042
2021-08-11 16:30:05.661 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:30:05.865 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:30:05.865 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on cassandra/10.242.110.137:9042
2021-08-11 16:30:05.875 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Successful reconnection to cassandra/10.242.110.137:9042, setting host UP
2021-08-11 16:30:05.875 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 16:30:05.875 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:30:05.876 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 16:30:06.163 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:30:06.164 DEBUG 10 --- [-reconnection-1] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:30:06.164 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 16:30:06.164 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:30:33.580 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:30:33.582 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:30:35.878 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:30:35.880 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:30:36.165 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:30:36.168 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:31:03.609 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:31:03.611 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:31:05.906 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:31:05.908 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:31:35.934 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:31:35.937 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:32:05.965 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:32:05.967 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:32:08.775 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:32:08.777 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:32:35.995 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:32:35.997 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:32:38.805 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:32:38.807 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:33:06.021 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:33:06.023 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:33:08.909 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:33:08.911 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:33:36.049 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:33:36.052 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:33:53.993 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:33:53.996 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:34:06.054 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:34:06.056 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:34:36.057 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:34:36.060 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:34:44.097 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:34:44.100 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:34:49.125 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=1, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:34:49.127 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=1, closed=false] heartbeat query succeeded
2021-08-11 16:35:06.085 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:35:06.087 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:35:36.090 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:35:36.092 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:35:49.190 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:35:49.192 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:36:06.117 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:36:06.120 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:36:19.234 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:36:19.237 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:36:36.146 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:36:36.149 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:36:59.347 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 16:36:59.347 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 16:36:59.360 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-21, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-20, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-3] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 0ms
2021-08-11 16:36:59.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [Control connection] starting reconnection attempt
2021-08-11 16:36:59.362 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-21, inFlight=0, closed=true] closing connection
2021-08-11 16:36:59.362 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:36:59.362 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-20, inFlight=0, closed=true] closing connection
2021-08-11 16:36:59.363 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 16:36:59.363 DEBUG 10 --- [uster-worker-10] c.d.driver.core.ControlConnection : [Control connection] /10.242.184.131:9042 is down/removed and it was the control host, triggering reconnect
2021-08-11 16:36:59.363 DEBUG 10 --- [uster-worker-10] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:36:59.363 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 16:36:59.364 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-23, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:37:00.348 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:37:00.364 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:00.366 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-22, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:37:00.366 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-22, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:37:00.366 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-22, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:00.366 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-22, inFlight=0, closed=true] closing connection
2021-08-11 16:37:00.366 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:37:00.366 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:37:00.366 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 16:37:00.473 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:37:00.473 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:37:00.473 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 16:37:00.473 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:37:00.473 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-22, inFlight=0, closed=true] closing connection
2021-08-11 16:37:00.475 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-23, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.buffer.WrappedByteBuf.writeBytes(WrappedByteBuf.java:821)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:37:00.475 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-23, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:37:00.475 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-23, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:00.475 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-23, inFlight=0, closed=true] closing connection
2021-08-11 16:37:00.475 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Not terminating Connection[cassandra/10.242.110.137:9042-23, inFlight=0, closed=true]: there are still pending requests
2021-08-11 16:37:00.475 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 16:37:00.476 DEBUG 10 --- [uster-worker-11] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:37:00.476 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 16:37:00.476 DEBUG 10 --- [-reconnection-0] c.d.driver.core.ControlConnection : [Control connection] error on cassandra/10.242.110.137:9042 connection, trying next host
com.datastax.driver.core.exceptions.TransportException: [cassandra/10.242.110.137:9042] Connection has been closed
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1397)
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1378)
at com.datastax.driver.core.Connection.defunct(Connection.java:570)
at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1301)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:271)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:37:00.476 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:37:00.478 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-20, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:37:00.481 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-20, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:00.481 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:37:00.481 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-20, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:37:00.481 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-20, inFlight=0, closed=true] closing connection
2021-08-11 16:37:00.482 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-20, inFlight=0, closed=true] has already terminated
2021-08-11 16:37:00.482 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:00.482 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 1000 ms
2021-08-11 16:37:01.476 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:01.478 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-24, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:37:01.478 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-24, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:37:01.478 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-24, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:01.478 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-24, inFlight=0, closed=true] closing connection
2021-08-11 16:37:01.478 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:37:01.479 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:37:01.479 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 16:37:01.482 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:37:01.484 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-21, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:37:01.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-21, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:01.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:37:01.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-21, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:37:01.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-21, inFlight=0, closed=true] closing connection
2021-08-11 16:37:01.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-21, inFlight=0, closed=true] has already terminated
2021-08-11 16:37:01.487 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:01.487 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 2000 ms
2021-08-11 16:37:02.367 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:02.369 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-23, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:37:02.369 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-23, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:37:02.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-23, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:02.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-23, inFlight=0, closed=true] closing connection
2021-08-11 16:37:02.370 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:37:02.370 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:37:02.370 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 16:37:03.479 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:03.482 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-25, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:37:03.482 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-25, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:37:03.482 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-25, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:03.482 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-25, inFlight=0, closed=true] closing connection
2021-08-11 16:37:03.482 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:37:03.483 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:37:03.483 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 16:37:03.487 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:37:03.488 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-22, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:37:03.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-22, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:03.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:37:03.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-22, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:37:03.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-22, inFlight=0, closed=true] closing connection
2021-08-11 16:37:03.493 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-22, inFlight=0, closed=true] has already terminated
2021-08-11 16:37:03.493 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:03.493 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 4000 ms
2021-08-11 16:37:04.466 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [850666576-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33d6eed2 at consistency null
2021-08-11 16:37:04.466 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [850666576-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:04.519 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [439073476-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@253221de at consistency null
2021-08-11 16:37:04.520 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [439073476-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:04.622 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1779689902-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@e501105 at consistency null
2021-08-11 16:37:04.622 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1779689902-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:04.825 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [350371217-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d9c8612 at consistency null
2021-08-11 16:37:04.825 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [350371217-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:05.228 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [124194297-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@19384b0d at consistency null
2021-08-11 16:37:05.228 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [124194297-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:06.031 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1079863883-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2db0b525 at consistency null
2021-08-11 16:37:06.032 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1079863883-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:06.371 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:06.373 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-24, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:37:06.373 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-24, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:37:06.373 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-24, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:06.373 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-24, inFlight=0, closed=true] closing connection
2021-08-11 16:37:06.374 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:37:06.374 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:37:06.374 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 16:37:06.835 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [896972151-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@236651af at consistency null
2021-08-11 16:37:06.835 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [896972151-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:07.483 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:07.485 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-26, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:37:07.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-26, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:37:07.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-26, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:07.486 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:37:07.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-26, inFlight=0, closed=true] closing connection
2021-08-11 16:37:07.486 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:37:07.486 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 16:37:07.493 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:37:07.494 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-23, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:37:07.497 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-23, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:07.497 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:37:07.497 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-23, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:37:07.497 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-23, inFlight=0, closed=true] closing connection
2021-08-11 16:37:07.498 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-23, inFlight=0, closed=true] has already terminated
2021-08-11 16:37:07.498 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:07.498 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 8000 ms
2021-08-11 16:37:08.438 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [813504765-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29392336 at consistency null
2021-08-11 16:37:08.438 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [813504765-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:08.440 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:36:58.865Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:37:13.444 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [256558030-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38c3f497 at consistency null
2021-08-11 16:37:13.444 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [256558030-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:13.496 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [611250014-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7dcbf99f at consistency null
2021-08-11 16:37:13.497 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [611250014-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:13.599 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [615721897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@393a1cc6 at consistency null
2021-08-11 16:37:13.599 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [615721897-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:13.703 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2085713202-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@597aa844 at consistency null
2021-08-11 16:37:13.703 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2085713202-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:13.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [635211205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7eba88b0 at consistency null
2021-08-11 16:37:13.806 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [635211205-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:14.009 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1981514303-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c9f91f9 at consistency null
2021-08-11 16:37:14.009 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1981514303-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:14.214 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1271402806-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35aa7a80 at consistency null
2021-08-11 16:37:14.214 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1271402806-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:14.375 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:14.377 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-25, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:37:14.377 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-25, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:37:14.377 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-25, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:14.377 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-25, inFlight=0, closed=true] closing connection
2021-08-11 16:37:14.378 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:37:14.378 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:37:14.378 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 16:37:14.617 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1011291544-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@391eccf9 at consistency null
2021-08-11 16:37:14.617 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1011291544-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:15.420 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2100565360-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@13cb2c62 at consistency null
2021-08-11 16:37:15.420 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2100565360-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:15.487 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:15.488 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-27, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:37:15.489 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-27, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:37:15.489 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-27, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:15.489 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-27, inFlight=0, closed=true] closing connection
2021-08-11 16:37:15.489 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:37:15.489 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:37:15.489 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 16:37:15.498 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:37:15.500 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-24, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:37:15.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-24, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:15.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:37:15.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-24, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:37:15.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-24, inFlight=0, closed=true] closing connection
2021-08-11 16:37:15.502 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:15.502 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 16000 ms
2021-08-11 16:37:15.502 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-24, inFlight=0, closed=true] has already terminated
2021-08-11 16:37:17.023 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1460955795-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ad5027f at consistency null
2021-08-11 16:37:17.023 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1460955795-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:17.024 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:36:58.865Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:37:22.028 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1558085862-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@50b7857b at consistency null
2021-08-11 16:37:22.028 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1558085862-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.056 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1488037168-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38996679 at consistency null
2021-08-11 16:37:22.056 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1488037168-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.108 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [99853507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@769d3ec1 at consistency null
2021-08-11 16:37:22.108 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [99853507-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.161 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1900480028-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44691c97 at consistency null
2021-08-11 16:37:22.161 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1900480028-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.213 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1505124706-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2819c53c at consistency null
2021-08-11 16:37:22.214 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1505124706-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.316 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [217831233-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b40c4e8 at consistency null
2021-08-11 16:37:22.316 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [217831233-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.419 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [339128225-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44348277 at consistency null
2021-08-11 16:37:22.419 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [339128225-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1560816861-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4835b3c2 at consistency null
2021-08-11 16:37:22.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1560816861-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.650 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [929037167-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ee3afdd at consistency null
2021-08-11 16:37:22.650 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [929037167-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:22.853 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1699504922-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7359596d at consistency null
2021-08-11 16:37:22.853 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1699504922-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:23.056 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1143374773-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@131e4a4b at consistency null
2021-08-11 16:37:23.056 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1143374773-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:23.258 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1490297128-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@76504049 at consistency null
2021-08-11 16:37:23.258 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1490297128-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:23.461 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [278412766-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7c552fea at consistency null
2021-08-11 16:37:23.461 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [278412766-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:23.864 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [382392420-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1df86e39 at consistency null
2021-08-11 16:37:23.864 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [382392420-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:24.267 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1719781923-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69de37c7 at consistency null
2021-08-11 16:37:24.267 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1719781923-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:25.070 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [522549946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@583128a9 at consistency null
2021-08-11 16:37:25.070 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [522549946-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:25.873 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1317192269-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d4e663a at consistency null
2021-08-11 16:37:25.873 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1317192269-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:25.874 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:36:58.865Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:37:30.378 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:30.381 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-26, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:37:30.381 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-26, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:37:30.381 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-26, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:30.381 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-26, inFlight=0, closed=true] closing connection
2021-08-11 16:37:30.381 DEBUG 10 --- [uster-worker-11] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:37:30.381 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:37:30.382 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 16:37:30.878 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [248195585-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5319580c at consistency null
2021-08-11 16:37:30.878 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [248195585-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:30.906 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1718827305-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4be13da7 at consistency null
2021-08-11 16:37:30.906 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1718827305-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:30.934 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1387757559-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1ac43012 at consistency null
2021-08-11 16:37:30.934 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1387757559-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:30.987 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [184508230-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6445eac6 at consistency null
2021-08-11 16:37:30.987 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [184508230-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:31.090 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1572071226-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7d05077a at consistency null
2021-08-11 16:37:31.090 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1572071226-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:31.245 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-23, inFlight=0, closed=true] has already terminated
2021-08-11 16:37:31.293 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [578172884-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3fcb1178 at consistency null
2021-08-11 16:37:31.293 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [578172884-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:31.489 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:37:31.491 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-28, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:37:31.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-28, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:37:31.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-28, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:37:31.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-28, inFlight=0, closed=true] closing connection
2021-08-11 16:37:31.492 DEBUG 10 --- [uster-worker-10] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:37:31.493 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:37:31.493 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 16:37:31.495 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1264037034-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@434f233 at consistency null
2021-08-11 16:37:31.495 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1264037034-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:31.503 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:37:31.504 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-25, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:37:31.506 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-25, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:31.506 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:37:31.506 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-25, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:37:31.506 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-25, inFlight=0, closed=true] closing connection
2021-08-11 16:37:31.506 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-25, inFlight=0, closed=true] has already terminated
2021-08-11 16:37:31.506 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:37:31.506 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 32000 ms
2021-08-11 16:37:31.897 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1000529897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f6c2ba9 at consistency null
2021-08-11 16:37:31.898 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1000529897-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:32.700 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2037015952-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a87dbe9 at consistency null
2021-08-11 16:37:32.700 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2037015952-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:34.303 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1789998590-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7da04910 at consistency null
2021-08-11 16:37:34.304 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1789998590-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:34.305 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:36:58.865Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:37:39.347 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [485881598-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e2689fd at consistency null
2021-08-11 16:37:39.347 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [485881598-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:39.447 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [830816030-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70b9d0f3 at consistency null
2021-08-11 16:37:39.447 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [830816030-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:39.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [468294235-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54c5059e at consistency null
2021-08-11 16:37:39.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [468294235-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:39.600 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2014593565-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2b523edf at consistency null
2021-08-11 16:37:39.600 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2014593565-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:39.653 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1273305575-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f514db8 at consistency null
2021-08-11 16:37:39.653 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1273305575-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:39.755 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1400939082-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3444e39 at consistency null
2021-08-11 16:37:39.755 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1400939082-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:39.858 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [82628899-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39da2c48 at consistency null
2021-08-11 16:37:39.858 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [82628899-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:39.961 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1588756233-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71da0831 at consistency null
2021-08-11 16:37:39.961 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1588756233-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:40.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1039892249-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@493b73f at consistency null
2021-08-11 16:37:40.064 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1039892249-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:40.267 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1799297054-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25870412 at consistency null
2021-08-11 16:37:40.268 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1799297054-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:40.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [154712467-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dd7d486 at consistency null
2021-08-11 16:37:40.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [154712467-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:40.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1153418634-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cedd68d at consistency null
2021-08-11 16:37:40.950 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1153418634-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:41.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [576350643-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@755ebe60 at consistency null
2021-08-11 16:37:41.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [576350643-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:42.555 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [825653774-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68e4934d at consistency null
2021-08-11 16:37:42.556 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [825653774-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:37:42.557 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:36:58.865Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:38:02.382 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:38:02.385 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:38:03.461 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:38:03.462 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Successful reconnection to /10.242.184.131:9042, setting host UP
2021-08-11 16:38:03.462 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 16:38:03.462 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:38:03.462 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 16:38:03.493 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:38:03.495 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:38:03.554 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:38:03.554 DEBUG 10 --- [-reconnection-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:38:03.554 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 16:38:03.555 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:38:03.555 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:38:03.557 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:38:04.146 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:38:04.147 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2021-08-11 16:38:04.158 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:38:04.166 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 8 milliseconds
2021-08-11 16:38:04.179 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2021-08-11 16:38:04.371 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:38:04.375 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 4 milliseconds
2021-08-11 16:38:04.381 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] established to /10.242.184.131:9042
2021-08-11 16:38:04.381 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:38:14.964 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:38:14.965 DEBUG 10 --- [-reconnection-0] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on cassandra/10.242.110.137:9042
2021-08-11 16:38:15.049 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Successful reconnection to cassandra/10.242.110.137:9042, setting host UP
2021-08-11 16:38:15.050 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 16:38:15.050 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:38:15.050 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 16:38:15.370 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:38:15.370 DEBUG 10 --- [-reconnection-0] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:38:15.370 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 16:38:15.370 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:38:45.062 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:38:45.065 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:38:52.675 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:38:52.676 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:39:07.696 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:39:07.699 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:39:15.093 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:39:15.095 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:39:37.777 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:39:37.779 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:39:45.122 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:39:45.124 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:39:52.813 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:39:52.860 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:40:15.125 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:40:15.128 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:40:22.949 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:40:22.951 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:40:42.984 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:40:42.987 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:40:45.157 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:40:45.159 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:41:15.177 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:41:15.180 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:41:38.195 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:41:38.203 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:41:45.207 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:41:45.209 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:42:03.265 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:42:03.267 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:42:13.268 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:42:13.270 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:42:15.225 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:42:15.228 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:42:45.234 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:42:45.236 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:42:48.304 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:42:48.306 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:42:58.324 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:42:58.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:43:15.264 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:43:15.266 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:43:43.385 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:43:43.388 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:43:45.293 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:43:45.295 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:44:15.324 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:44:15.326 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:44:28.490 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:44:28.493 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:44:45.355 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:44:45.358 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:44:59.346 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 16:44:59.346 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 16:44:59.460 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:44:59.460 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:44:59.460 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:44:59.460 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:44:59.460 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 16:44:59.460 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-27, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:44:59.460 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-28, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:44:59.461 DEBUG 10 --- [er-nio-worker-2] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 0ms
2021-08-11 16:44:59.461 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [Control connection] starting reconnection attempt
2021-08-11 16:44:59.461 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-28, inFlight=0, closed=true] closing connection
2021-08-11 16:44:59.462 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-27, inFlight=0, closed=true] closing connection
2021-08-11 16:44:59.462 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:44:59.462 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 16:44:59.462 DEBUG 10 --- [uster-worker-12] c.d.driver.core.ControlConnection : [Control connection] /10.242.184.131:9042 is down/removed and it was the control host, triggering reconnect
2021-08-11 16:44:59.463 DEBUG 10 --- [uster-worker-12] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:44:59.463 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 16:44:59.465 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-30, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:45:00.268 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-30, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:45:00.268 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-30, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:45:00.269 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 16:45:00.269 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-30, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:45:00.269 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-30, inFlight=0, closed=true] closing connection
2021-08-11 16:45:00.269 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Not terminating Connection[cassandra/10.242.110.137:9042-30, inFlight=0, closed=true]: there are still pending requests
2021-08-11 16:45:00.270 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:45:00.270 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] error on cassandra/10.242.110.137:9042 connection, trying next host
com.datastax.driver.core.exceptions.TransportException: [cassandra/10.242.110.137:9042] Connection has been closed
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1397)
at com.datastax.driver.core.Connection$ConnectionCloseFuture.force(Connection.java:1378)
at com.datastax.driver.core.Connection.defunct(Connection.java:570)
at com.datastax.driver.core.Connection$Dispatcher.exceptionCaught(Connection.java:1301)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279)
at io.netty.channel.AbstractChannelHandlerContext.fireExceptionCaught(AbstractChannelHandlerContext.java:271)
at io.netty.channel.DefaultChannelPipeline$HeadContext.exceptionCaught(DefaultChannelPipeline.java:1377)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:300)
at io.netty.channel.AbstractChannelHandlerContext.invokeExceptionCaught(AbstractChannelHandlerContext.java:279)
at io.netty.channel.DefaultChannelPipeline.fireExceptionCaught(DefaultChannelPipeline.java:907)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.handleReadException(AbstractNioByteChannel.java:125)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:174)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:45:00.270 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:45:00.270 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:45:00.270 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:00.271 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-29, inFlight=0, closed=true] closing connection
2021-08-11 16:45:00.271 DEBUG 10 --- [uster-worker-13] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 16:45:00.272 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-26, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:45:00.272 DEBUG 10 --- [uster-worker-13] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:45:00.273 DEBUG 10 --- [uster-worker-13] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 16:45:00.274 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-26, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:00.274 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:45:00.275 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-26, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:45:00.275 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-26, inFlight=0, closed=true] closing connection
2021-08-11 16:45:00.275 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-26, inFlight=0, closed=true] has already terminated
2021-08-11 16:45:00.275 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:00.275 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 1000 ms
2021-08-11 16:45:00.347 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:45:00.464 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:00.467 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-29, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:45:00.467 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-29, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:45:00.467 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-29, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:00.467 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-29, inFlight=0, closed=true] closing connection
2021-08-11 16:45:00.467 DEBUG 10 --- [uster-worker-13] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:45:00.468 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:45:00.468 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 16:45:01.273 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:01.275 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-31, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:45:01.275 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:45:01.275 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-31, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:45:01.275 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-31, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:01.276 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-31, inFlight=0, closed=true] closing connection
2021-08-11 16:45:01.276 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:45:01.276 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:45:01.276 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 16:45:01.277 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-27, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:45:01.280 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-27, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:01.280 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:45:01.280 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-27, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:45:01.280 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-27, inFlight=0, closed=true] closing connection
2021-08-11 16:45:01.281 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-27, inFlight=0, closed=true] has already terminated
2021-08-11 16:45:01.281 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:01.281 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 2000 ms
2021-08-11 16:45:02.468 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:02.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-30, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:45:02.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-30, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:45:02.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-30, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:02.470 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-30, inFlight=0, closed=true] closing connection
2021-08-11 16:45:02.471 DEBUG 10 --- [uster-worker-13] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:45:02.471 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:45:02.471 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 16:45:03.276 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:03.278 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-32, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:45:03.279 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-32, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:45:03.279 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-32, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:03.279 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-32, inFlight=0, closed=true] closing connection
2021-08-11 16:45:03.279 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:45:03.279 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:45:03.279 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 16:45:03.281 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:45:03.283 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-28, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:45:03.285 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-28, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:03.285 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:45:03.285 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-28, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:45:03.285 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-28, inFlight=0, closed=true] closing connection
2021-08-11 16:45:03.286 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-28, inFlight=0, closed=true] has already terminated
2021-08-11 16:45:03.286 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:03.286 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 4000 ms
2021-08-11 16:45:03.650 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1576305371-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@85c0925 at consistency null
2021-08-11 16:45:03.650 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1576305371-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:03.704 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [936638209-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@20bdc807 at consistency null
2021-08-11 16:45:03.704 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [936638209-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:03.807 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1637190938-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f510d3 at consistency null
2021-08-11 16:45:03.807 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1637190938-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:04.011 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [86855112-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6684439 at consistency null
2021-08-11 16:45:04.011 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [86855112-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:04.414 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2083976684-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e1570f0 at consistency null
2021-08-11 16:45:04.414 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2083976684-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:05.217 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1425374868-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@202670c5 at consistency null
2021-08-11 16:45:05.218 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1425374868-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:06.471 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:06.473 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-31, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:45:06.474 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-31, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:45:06.474 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-31, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:06.474 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-31, inFlight=0, closed=true] closing connection
2021-08-11 16:45:06.474 DEBUG 10 --- [uster-worker-13] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:45:06.474 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:45:06.474 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 16:45:06.821 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [424006733-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ffa9f9a at consistency null
2021-08-11 16:45:06.821 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [424006733-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:06.823 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:44:58.074Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:45:07.280 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:07.282 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-33, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:45:07.282 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-33, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:45:07.282 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-33, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:07.282 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-33, inFlight=0, closed=true] closing connection
2021-08-11 16:45:07.282 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:45:07.283 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:45:07.283 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 16:45:07.286 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:45:07.288 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-29, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:45:07.291 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-29, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:07.291 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:45:07.291 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-29, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:45:07.291 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-29, inFlight=0, closed=true] closing connection
2021-08-11 16:45:07.292 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-29, inFlight=0, closed=true] has already terminated
2021-08-11 16:45:07.292 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:07.292 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 8000 ms
2021-08-11 16:45:11.826 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1986632096-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4bf75bd0 at consistency null
2021-08-11 16:45:11.827 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1986632096-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:11.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1035705108-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ea030ae at consistency null
2021-08-11 16:45:11.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1035705108-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:11.933 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1111835625-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d93d9e7 at consistency null
2021-08-11 16:45:11.933 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1111835625-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:12.036 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1972699216-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@464e9380 at consistency null
2021-08-11 16:45:12.036 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1972699216-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:12.240 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1326130139-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37967942 at consistency null
2021-08-11 16:45:12.240 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1326130139-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:12.644 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [780755477-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@383c1a5e at consistency null
2021-08-11 16:45:12.644 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [780755477-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:13.047 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1178285606-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3be3b05f at consistency null
2021-08-11 16:45:13.047 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1178285606-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:13.851 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [468793649-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@983f630 at consistency null
2021-08-11 16:45:13.851 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [468793649-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:14.475 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:14.477 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-32, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:45:14.477 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-32, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:45:14.477 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-32, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:14.477 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-32, inFlight=0, closed=true] closing connection
2021-08-11 16:45:14.478 DEBUG 10 --- [uster-worker-13] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:45:14.478 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:45:14.478 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 16:45:14.654 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [225625199-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@73aaf49e at consistency null
2021-08-11 16:45:14.654 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [225625199-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:14.656 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:44:58.074Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:45:15.283 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:15.285 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-34, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:45:15.285 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-34, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:45:15.286 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-34, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:15.286 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-34, inFlight=0, closed=true] closing connection
2021-08-11 16:45:15.286 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:45:15.286 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:45:15.286 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 16:45:15.292 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:45:15.294 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-30, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:45:15.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-30, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:15.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:45:15.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-30, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:45:15.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-30, inFlight=0, closed=true] closing connection
2021-08-11 16:45:15.296 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-30, inFlight=0, closed=true] has already terminated
2021-08-11 16:45:15.296 ERROR 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:15.297 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 16000 ms
2021-08-11 16:45:19.659 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1754175006-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@650ebead at consistency null
2021-08-11 16:45:19.659 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1754175006-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:19.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1998264111-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3617e2ad at consistency null
2021-08-11 16:45:19.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1998264111-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:19.739 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1697747020-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@d78c72c at consistency null
2021-08-11 16:45:19.740 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1697747020-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:19.842 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2130407737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2952cc19 at consistency null
2021-08-11 16:45:19.842 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2130407737-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:20.045 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1658402462-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3939e448 at consistency null
2021-08-11 16:45:20.045 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1658402462-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:20.448 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1305798533-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30b27b48 at consistency null
2021-08-11 16:45:20.448 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1305798533-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:21.251 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [613330307-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b389bd5 at consistency null
2021-08-11 16:45:21.251 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [613330307-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:22.054 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [31027858-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44b5b2af at consistency null
2021-08-11 16:45:22.054 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [31027858-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:23.657 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [715114993-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32d805d5 at consistency null
2021-08-11 16:45:23.658 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [715114993-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:23.659 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:44:58.074Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:45:28.665 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1084095464-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@111350fd at consistency null
2021-08-11 16:45:28.665 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1084095464-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:28.747 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [296658125-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@35073b1d at consistency null
2021-08-11 16:45:28.747 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [296658125-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:28.850 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1537808776-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e9801f8 at consistency null
2021-08-11 16:45:28.850 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1537808776-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:29.053 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [254645069-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@44113a4d at consistency null
2021-08-11 16:45:29.053 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [254645069-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:29.255 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [549485563-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@180a350a at consistency null
2021-08-11 16:45:29.256 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [549485563-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:29.659 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1152138210-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@14593ac5 at consistency null
2021-08-11 16:45:29.659 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1152138210-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:30.061 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1465524134-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ab07e77 at consistency null
2021-08-11 16:45:30.062 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1465524134-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:30.464 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [873355771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@27bef23e at consistency null
2021-08-11 16:45:30.464 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [873355771-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:30.479 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:30.481 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-33, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:45:30.481 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-33, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:45:30.482 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-33, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:30.482 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-33, inFlight=0, closed=true] closing connection
2021-08-11 16:45:30.482 DEBUG 10 --- [uster-worker-13] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:45:30.482 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:45:30.482 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 16:45:30.867 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [389215010-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e5531a6 at consistency null
2021-08-11 16:45:30.867 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [389215010-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:31.253 DEBUG 10 --- [ection-reaper-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-30, inFlight=0, closed=true] has already terminated
2021-08-11 16:45:31.286 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:45:31.288 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-35, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:45:31.289 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-35, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:45:31.289 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-35, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:45:31.289 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-35, inFlight=0, closed=true] closing connection
2021-08-11 16:45:31.289 DEBUG 10 --- [uster-worker-12] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:45:31.289 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:45:31.289 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 16:45:31.297 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:45:31.298 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-31, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:45:31.301 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.154.72:9042-31, inFlight=0, closed=false] because: Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:31.301 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preventing new connections for the next 1000 ms
2021-08-11 16:45:31.301 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-31, inFlight=0, closed=false] failed, remaining = 1
2021-08-11 16:45:31.301 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-31, inFlight=0, closed=true] closing connection
2021-08-11 16:45:31.302 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-31, inFlight=0, closed=true] has already terminated
2021-08-11 16:45:31.302 ERROR 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Authentication error on host /10.242.154.72:9042: Error during authentication of user cassandra : org.apache.cassandra.exceptions.UnavailableException: Cannot achieve consistency level QUORUM
2021-08-11 16:45:31.302 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [Control connection] next reconnection attempt in 32000 ms
2021-08-11 16:45:31.670 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [853850381-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@a520413 at consistency null
2021-08-11 16:45:31.670 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [853850381-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:32.473 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [902051800-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d4dccfe at consistency null
2021-08-11 16:45:32.473 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [902051800-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:32.474 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:44:58.074Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:45:37.478 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1942184218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43c93695 at consistency null
2021-08-11 16:45:37.478 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1942184218-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:37.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1936205731-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70447264 at consistency null
2021-08-11 16:45:37.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1936205731-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:37.600 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1683215378-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a2c92e3 at consistency null
2021-08-11 16:45:37.600 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1683215378-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:37.653 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1891456439-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@49258964 at consistency null
2021-08-11 16:45:37.653 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1891456439-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:37.705 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [404631819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4352634b at consistency null
2021-08-11 16:45:37.706 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [404631819-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:37.808 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [712591508-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@10d6ee17 at consistency null
2021-08-11 16:45:37.808 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [712591508-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:38.050 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1349577357-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e63dd64 at consistency null
2021-08-11 16:45:38.050 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1349577357-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:38.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1249517017-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b681965 at consistency null
2021-08-11 16:45:38.454 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1249517017-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:38.857 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [595979252-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2bf06578 at consistency null
2021-08-11 16:45:38.857 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [595979252-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:39.260 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1138244344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@cf18047 at consistency null
2021-08-11 16:45:39.260 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1138244344-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:40.062 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [338400396-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c79f440 at consistency null
2021-08-11 16:45:40.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [338400396-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:40.865 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [237356805-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37e1679b at consistency null
2021-08-11 16:45:40.866 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [237356805-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:45:40.867 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:44:58.074Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:46:02.483 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:46:02.484 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:46:03.289 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:46:03.291 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:46:04.650 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:46:04.651 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Successful reconnection to /10.242.184.131:9042, setting host UP
2021-08-11 16:46:04.651 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 16:46:04.651 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:46:04.651 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 16:46:04.848 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:46:04.849 DEBUG 10 --- [-reconnection-1] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:46:04.849 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 16:46:04.849 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:46:04.849 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] preparing to open 1 new connections, total = 2
2021-08-11 16:46:04.851 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:46:04.964 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:46:04.964 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Successful reconnection to cassandra/10.242.110.137:9042, setting host UP
2021-08-11 16:46:04.964 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 16:46:04.965 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:46:04.965 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 16:46:04.980 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:46:04.980 DEBUG 10 --- [-reconnection-0] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:46:04.980 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 16:46:04.980 DEBUG 10 --- [-reconnection-0] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:46:05.049 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.154.72:9042] Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:46:05.049 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing node list and token map
2021-08-11 16:46:05.055 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:46:05.061 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 5 milliseconds
2021-08-11 16:46:05.069 DEBUG 10 --- [-reconnection-1] c.d.driver.core.ControlConnection : [Control connection] Refreshing schema
2021-08-11 16:46:05.103 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Computing token to replica map for keyspace: sdctitan.
2021-08-11 16:46:05.107 DEBUG 10 --- [-reconnection-1] licationStrategy$NetworkTopologyStrategy : Token to replica map computation for keyspace sdctitan completed in 4 milliseconds
2021-08-11 16:46:05.112 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [Control connection] established to /10.242.154.72:9042
2021-08-11 16:46:05.112 DEBUG 10 --- [-reconnection-1] c.d.d.core.AbstractReconnectionHandler : Reconnection successful, cleared the future
2021-08-11 16:46:35.081 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:46:35.083 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:47:05.110 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:47:05.112 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:47:21.095 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:47:21.097 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:47:35.125 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:47:35.127 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:47:36.114 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:47:36.116 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:48:05.156 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:48:05.158 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:48:06.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:48:06.185 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:48:35.187 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:48:35.190 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:48:51.271 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:48:51.274 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:49:05.217 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:49:05.219 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:49:16.298 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:49:16.301 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:49:35.245 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:49:35.247 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:50:05.253 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:50:05.255 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:50:26.688 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:50:26.690 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:50:35.278 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:50:35.280 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:51:05.309 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:51:05.311 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:51:11.729 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:51:11.732 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:51:35.337 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:51:35.339 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:52:01.905 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:52:01.907 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:52:05.355 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:52:05.356 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:52:35.383 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:52:35.385 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:52:59.549 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:52:59.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:52:59.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 16:52:59.550 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-34, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:52:59.551 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-34, inFlight=0, closed=true] closing connection
2021-08-11 16:52:59.551 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 16:52:59.553 DEBUG 10 --- [uster-worker-14] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:52:59.553 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 16:52:59.651 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 16:52:59.651 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 16:53:00.174 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 16:53:00.174 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 16:53:00.174 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 16:53:00.174 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:00.175 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-36, inFlight=0, closed=true] closing connection
2021-08-11 16:53:00.175 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 16:53:00.175 DEBUG 10 --- [uster-worker-15] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 16:53:00.175 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 16:53:00.273 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.110.137:9042, scheduling delivery
2021-08-11 16:53:00.273 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event DOWN
2021-08-11 16:53:00.553 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:00.556 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-35, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:53:00.556 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-35, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:53:00.556 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-35, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:00.556 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-35, inFlight=0, closed=true] closing connection
2021-08-11 16:53:00.556 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:53:00.556 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:53:00.556 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 16:53:01.178 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:01.179 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-37, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:53:01.180 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-37, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:53:01.180 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-37, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:01.180 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-37, inFlight=0, closed=true] closing connection
2021-08-11 16:53:01.180 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 16:53:01.180 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 16:53:01.180 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:53:01.274 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:53:01.274 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:53:02.055 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1945879032-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d003a3f at consistency null
2021-08-11 16:53:02.055 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1945879032-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:02.083 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1648564765-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d2a0f69 at consistency null
2021-08-11 16:53:02.083 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1648564765-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:02.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1537788797-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@638d3140 at consistency null
2021-08-11 16:53:02.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1537788797-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:02.137 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [642422565-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15ca249c at consistency null
2021-08-11 16:53:02.137 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [642422565-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:02.190 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [185824789-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79f725d8 at consistency null
2021-08-11 16:53:02.190 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [185824789-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:02.292 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1903826584-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@64bb3cc8 at consistency null
2021-08-11 16:53:02.292 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1903826584-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:02.495 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1813305577-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6eb6062e at consistency null
2021-08-11 16:53:02.495 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1813305577-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:02.557 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:02.558 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-36, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:53:02.558 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-36, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:53:02.558 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-36, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:02.559 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-36, inFlight=0, closed=true] closing connection
2021-08-11 16:53:02.559 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:53:02.559 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:53:02.559 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 16:53:02.698 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [504732145-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a9c21a0 at consistency null
2021-08-11 16:53:02.698 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [504732145-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:03.101 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [237434872-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6a8508ce at consistency null
2021-08-11 16:53:03.101 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [237434872-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:03.181 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:03.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-38, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:53:03.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-38, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:53:03.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-38, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:03.183 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-38, inFlight=0, closed=true] closing connection
2021-08-11 16:53:03.183 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:53:03.184 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 16:53:03.184 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 16:53:03.504 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [801762930-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5765242a at consistency null
2021-08-11 16:53:03.504 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [801762930-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:03.907 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1708173554-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@93ccf9e at consistency null
2021-08-11 16:53:03.907 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1708173554-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:04.311 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1463320488-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42e67e38 at consistency null
2021-08-11 16:53:04.311 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1463320488-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:05.114 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1855507328-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67cdd2 at consistency null
2021-08-11 16:53:05.114 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1855507328-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:05.117 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:52:56.547Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:53:06.559 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:06.561 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-37, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:53:06.561 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-37, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:53:06.561 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-37, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:06.562 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-37, inFlight=0, closed=true] closing connection
2021-08-11 16:53:06.562 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:53:06.562 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:53:06.562 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 16:53:07.184 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:07.187 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-39, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:53:07.187 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-39, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:53:07.187 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-39, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:07.188 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-39, inFlight=0, closed=true] closing connection
2021-08-11 16:53:07.188 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:53:07.188 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 16:53:07.188 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 16:53:10.121 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [239211103-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60143652 at consistency null
2021-08-11 16:53:10.122 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [239211103-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:10.149 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1857399901-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5e236edf at consistency null
2021-08-11 16:53:10.149 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1857399901-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:10.177 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2053393180-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f03a966 at consistency null
2021-08-11 16:53:10.177 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2053393180-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:10.230 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1384667931-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4a1b7031 at consistency null
2021-08-11 16:53:10.230 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1384667931-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:10.333 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1288076287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b7c6986 at consistency null
2021-08-11 16:53:10.333 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1288076287-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:10.536 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [257089918-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e6caba0 at consistency null
2021-08-11 16:53:10.536 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [257089918-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:10.939 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1471575843-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c767392 at consistency null
2021-08-11 16:53:10.939 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1471575843-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:11.742 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1198964301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@dfd12d5 at consistency null
2021-08-11 16:53:11.742 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1198964301-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:12.545 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [940584159-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@65257ab5 at consistency null
2021-08-11 16:53:12.545 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [940584159-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:13.348 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [550593458-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e3ffc1a at consistency null
2021-08-11 16:53:13.348 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [550593458-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:13.349 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:52:56.547Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:53:14.562 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:14.565 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-38, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:53:14.565 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-38, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:53:14.565 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-38, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:14.565 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-38, inFlight=0, closed=true] closing connection
2021-08-11 16:53:14.565 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:53:14.565 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:53:14.566 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 16:53:15.189 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:15.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-40, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:53:15.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-40, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:53:15.191 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-40, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:15.192 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-40, inFlight=0, closed=true] closing connection
2021-08-11 16:53:15.192 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:53:15.192 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 16:53:15.192 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 16:53:18.354 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1153046500-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@617aecac at consistency null
2021-08-11 16:53:18.354 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1153046500-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.382 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [919363048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f431ec at consistency null
2021-08-11 16:53:18.382 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [919363048-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.410 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [251918650-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4aca0315 at consistency null
2021-08-11 16:53:18.410 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [251918650-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [618977179-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7b24f97a at consistency null
2021-08-11 16:53:18.437 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [618977179-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.490 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [821424596-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4d8d653e at consistency null
2021-08-11 16:53:18.490 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [821424596-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.543 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1686296895-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@68d35632 at consistency null
2021-08-11 16:53:18.543 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1686296895-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.596 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1960322748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@721b3296 at consistency null
2021-08-11 16:53:18.596 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1960322748-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.699 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1979875251-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ea8ea86 at consistency null
2021-08-11 16:53:18.699 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1979875251-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:18.902 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1768311798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@8aacef8 at consistency null
2021-08-11 16:53:18.902 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1768311798-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:19.304 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [467614916-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7aba6b19 at consistency null
2021-08-11 16:53:19.305 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [467614916-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:19.707 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [780604613-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@511e695a at consistency null
2021-08-11 16:53:19.708 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [780604613-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:20.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1268586817-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@380b4525 at consistency null
2021-08-11 16:53:20.110 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1268586817-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:20.913 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [276812351-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@668881d2 at consistency null
2021-08-11 16:53:20.913 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [276812351-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:20.914 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:52:56.547Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:53:25.947 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1923409384-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4fc3dcb9 at consistency null
2021-08-11 16:53:25.947 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1923409384-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.047 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1679029829-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6d3538a2 at consistency null
2021-08-11 16:53:26.047 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1679029829-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.074 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1206365335-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ff56a4c at consistency null
2021-08-11 16:53:26.074 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1206365335-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.147 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [694640766-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2dd1fc71 at consistency null
2021-08-11 16:53:26.147 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [694640766-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.247 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2092518175-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7ca6db66 at consistency null
2021-08-11 16:53:26.247 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2092518175-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.349 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [123943699-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@38121c78 at consistency null
2021-08-11 16:53:26.349 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [123943699-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.452 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2107288700-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c11a63c at consistency null
2021-08-11 16:53:26.452 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2107288700-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.654 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1877759777-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e01e3bb at consistency null
2021-08-11 16:53:26.654 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1877759777-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:26.857 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1505429616-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@74b672f6 at consistency null
2021-08-11 16:53:26.857 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1505429616-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:27.260 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1193151479-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c6f41c4 at consistency null
2021-08-11 16:53:27.260 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1193151479-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:28.062 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1867490852-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4dde3c33 at consistency null
2021-08-11 16:53:28.063 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1867490852-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:29.665 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1645715891-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cf31208 at consistency null
2021-08-11 16:53:29.665 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1645715891-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:29.667 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:52:56.547Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:53:30.284 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:53:30.285 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:53:30.566 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:30.568 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-39, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 16:53:30.568 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-39, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 16:53:30.568 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-39, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:30.569 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-39, inFlight=0, closed=true] closing connection
2021-08-11 16:53:30.569 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 16:53:30.569 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:53:30.569 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 16:53:31.192 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:31.194 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-41, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 16:53:31.194 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-41, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 16:53:31.194 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-41, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 16:53:31.194 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-41, inFlight=0, closed=true] closing connection
2021-08-11 16:53:31.194 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 16:53:31.195 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 16:53:31.195 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 16:53:34.670 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1676598537-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21d17148 at consistency null
2021-08-11 16:53:34.670 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1676598537-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:34.723 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1419256624-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@45564428 at consistency null
2021-08-11 16:53:34.723 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1419256624-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:34.776 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1190449409-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51a2d2ce at consistency null
2021-08-11 16:53:34.776 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1190449409-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:34.829 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [422705749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b3110de at consistency null
2021-08-11 16:53:34.829 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [422705749-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:34.883 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [203586287-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71c071c1 at consistency null
2021-08-11 16:53:34.883 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [203586287-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:34.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [422988280-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c01cd5 at consistency null
2021-08-11 16:53:34.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [422988280-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:35.088 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2041245446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@41b39a61 at consistency null
2021-08-11 16:53:35.088 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2041245446-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:35.291 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1868513588-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12d7d569 at consistency null
2021-08-11 16:53:35.291 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1868513588-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:35.693 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1882565426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@606be08b at consistency null
2021-08-11 16:53:35.693 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1882565426-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:36.096 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1541887301-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bf690cd at consistency null
2021-08-11 16:53:36.097 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1541887301-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:36.900 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1973109728-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17267647 at consistency null
2021-08-11 16:53:36.900 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1973109728-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:38.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1624832646-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28f6b390 at consistency null
2021-08-11 16:53:38.547 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1624832646-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:38.548 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T16:52:56.547Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 16:53:43.551 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1030332820-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@15775c3f at consistency null
2021-08-11 16:53:43.552 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1030332820-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:43.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1556448769-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3eb81ffc at consistency null
2021-08-11 16:53:43.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1556448769-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:43.647 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1944996410-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@234f4188 at consistency null
2021-08-11 16:53:43.647 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1944996410-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:43.749 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1413845515-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c76677a at consistency null
2021-08-11 16:53:43.749 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1413845515-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:43.952 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [617675798-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5097ada6 at consistency null
2021-08-11 16:53:43.952 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [617675798-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:44.155 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1414894339-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e2e0fb7 at consistency null
2021-08-11 16:53:44.155 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1414894339-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:44.558 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1433119785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@77b448fb at consistency null
2021-08-11 16:53:44.558 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1433119785-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 16:53:55.851 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.184.131:9042, scheduling delivery
2021-08-11 16:53:55.851 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event UP
2021-08-11 16:53:56.852 DEBUG 10 --- [uster-worker-15] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.242.184.131:9042
2021-08-11 16:53:56.858 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 16:53:56.858 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:53:56.858 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 16:53:56.858 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:56.861 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:53:58.869 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.110.137:9042, scheduling delivery
2021-08-11 16:53:58.869 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event UP
2021-08-11 16:53:59.870 DEBUG 10 --- [uster-worker-14] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on cassandra/10.242.110.137:9042
2021-08-11 16:53:59.873 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 16:53:59.873 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 16:53:59.873 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 16:53:59.873 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 16:53:59.874 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 16:54:00.049 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:54:00.753 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:54:00.753 DEBUG 10 --- [uster-worker-15] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:54:00.753 DEBUG 10 --- [uster-worker-15] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 16:54:04.165 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 16:54:04.378 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 16:54:04.379 DEBUG 10 --- [uster-worker-14] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 16:54:04.379 DEBUG 10 --- [uster-worker-14] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 16:54:29.875 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:54:29.876 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:54:40.449 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:54:40.452 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:54:59.884 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:54:59.886 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:55:10.492 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:55:10.495 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:55:20.498 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:55:20.500 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:55:29.889 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:55:29.890 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:55:50.554 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:55:50.556 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:55:59.893 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:55:59.895 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:56:20.610 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:56:20.612 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:56:29.896 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:56:29.898 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:56:59.906 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:56:59.908 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:57:00.694 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:57:00.696 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:57:29.909 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:57:29.911 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:57:40.794 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:57:40.796 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:57:59.914 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:57:59.915 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:58:21.015 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:58:21.017 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:58:29.923 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:58:29.925 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:58:59.933 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:58:59.935 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:59:11.071 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:59:11.074 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:59:16.074 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:59:16.077 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:59:29.935 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:59:29.937 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 16:59:59.947 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 16:59:59.948 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:00:01.255 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:00:01.258 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:00:21.279 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:00:21.281 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:00:29.974 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:00:29.976 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:00:31.258 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:00:31.260 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:00:59.647 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 17:00:59.647 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 17:00:59.648 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 17:00:59.648 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-40, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:00:59.648 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-40, inFlight=0, closed=true] closing connection
2021-08-11 17:00:59.649 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 17:00:59.650 DEBUG 10 --- [uster-worker-16] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 17:00:59.650 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 17:00:59.656 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 17:00:59.656 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 17:00:59.971 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 17:00:59.971 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 17:00:59.971 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 17:00:59.971 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:00:59.972 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-42, inFlight=0, closed=true] closing connection
2021-08-11 17:00:59.972 DEBUG 10 --- [uster-worker-17] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 17:00:59.972 DEBUG 10 --- [uster-worker-17] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 17:00:59.972 DEBUG 10 --- [uster-worker-17] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 17:00:59.975 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.110.137:9042, scheduling delivery
2021-08-11 17:00:59.975 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event DOWN
2021-08-11 17:01:00.650 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:00.653 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-41, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:01:00.653 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-41, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:01:00.653 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-41, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:00.653 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-41, inFlight=0, closed=true] closing connection
2021-08-11 17:01:00.653 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:01:00.654 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 17:01:00.654 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 17:01:00.973 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:00.975 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-43, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:01:00.975 DEBUG 10 --- [uster-worker-17] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:01:00.975 DEBUG 10 --- [d-task-worker-0] com.datastax.driver.core.Cluster : Ignoring down event on cassandra/10.242.110.137:9042 because it still has active connections
2021-08-11 17:01:00.976 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-43, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:01:00.976 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-43, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:00.976 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-43, inFlight=0, closed=true] closing connection
2021-08-11 17:01:00.976 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:01:00.976 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 17:01:00.976 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 17:01:01.454 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [811669446-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@25d1c2f3 at consistency null
2021-08-11 17:01:01.454 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [811669446-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:01.507 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [87436133-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7bdc2249 at consistency null
2021-08-11 17:01:01.508 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [87436133-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:01.560 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [645636453-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2e7a4e59 at consistency null
2021-08-11 17:01:01.560 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [645636453-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:01.612 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [860294112-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5327c7fa at consistency null
2021-08-11 17:01:01.612 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [860294112-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:01.714 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [186709353-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@784a5137 at consistency null
2021-08-11 17:01:01.714 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [186709353-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:01.817 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1073318507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@621eea1d at consistency null
2021-08-11 17:01:01.817 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1073318507-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:02.020 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1148351608-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1eea6881 at consistency null
2021-08-11 17:01:02.020 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1148351608-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:02.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1331789597-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@739f74b1 at consistency null
2021-08-11 17:01:02.423 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1331789597-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:02.654 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:02.656 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-42, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:01:02.656 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-42, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:01:02.656 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-42, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:02.656 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-42, inFlight=0, closed=true] closing connection
2021-08-11 17:01:02.656 DEBUG 10 --- [uster-worker-17] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:01:02.657 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 17:01:02.657 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 17:01:02.825 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1130778661-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59f6401c at consistency null
2021-08-11 17:01:02.825 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1130778661-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:02.977 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:02.978 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-44, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:01:02.979 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-44, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:01:02.979 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-44, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:02.979 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-44, inFlight=0, closed=true] closing connection
2021-08-11 17:01:02.979 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:01:02.979 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 17:01:02.979 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 17:01:03.628 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [809443568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@97c163d at consistency null
2021-08-11 17:01:03.628 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [809443568-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:05.231 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1740198715-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@626141ae at consistency null
2021-08-11 17:01:05.231 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1740198715-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:05.233 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:00:55.947Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:01:06.657 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:06.659 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-43, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:01:06.659 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-43, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:01:06.659 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-43, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:06.659 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-43, inFlight=0, closed=true] closing connection
2021-08-11 17:01:06.659 DEBUG 10 --- [uster-worker-17] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:01:06.659 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 17:01:06.659 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 17:01:06.979 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:06.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-45, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:01:06.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-45, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:01:06.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-45, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:06.982 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-45, inFlight=0, closed=true] closing connection
2021-08-11 17:01:06.983 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:01:06.983 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 17:01:06.983 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 17:01:10.237 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [602161478-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46d1a630 at consistency null
2021-08-11 17:01:10.237 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [602161478-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.264 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1383258727-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@316a7aab at consistency null
2021-08-11 17:01:10.264 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1383258727-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [453199993-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@584e4f45 at consistency null
2021-08-11 17:01:10.317 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [453199993-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [906944681-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30ca09d3 at consistency null
2021-08-11 17:01:10.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [906944681-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [937523551-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7be1c9dc at consistency null
2021-08-11 17:01:10.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [937523551-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.475 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1938045990-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@193c4b29 at consistency null
2021-08-11 17:01:10.475 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1938045990-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.578 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1925571247-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@33a20b2c at consistency null
2021-08-11 17:01:10.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1925571247-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.681 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [865034230-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5346181a at consistency null
2021-08-11 17:01:10.682 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [865034230-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.785 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [261677913-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@261c9487 at consistency null
2021-08-11 17:01:10.785 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [261677913-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:10.988 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1563401088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d02e66e at consistency null
2021-08-11 17:01:10.988 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1563401088-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:11.190 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2284048-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7eda6d89 at consistency null
2021-08-11 17:01:11.191 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2284048-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:11.593 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1234916621-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d3f3c80 at consistency null
2021-08-11 17:01:11.593 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1234916621-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:12.396 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [22148428-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fc53b0c at consistency null
2021-08-11 17:01:12.396 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [22148428-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:13.999 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [545496445-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@153e870c at consistency null
2021-08-11 17:01:13.999 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [545496445-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:14.000 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:00:55.947Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:01:14.660 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:14.662 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-44, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:01:14.663 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-44, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:01:14.663 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-44, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:14.663 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-44, inFlight=0, closed=true] closing connection
2021-08-11 17:01:14.663 DEBUG 10 --- [uster-worker-17] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:01:14.663 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 17:01:14.663 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 17:01:14.983 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:14.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-46, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:01:14.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-46, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:01:14.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-46, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:14.985 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-46, inFlight=0, closed=true] closing connection
2021-08-11 17:01:14.986 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:01:14.986 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 17:01:14.986 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 17:01:19.005 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [921121063-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2af1bb7b at consistency null
2021-08-11 17:01:19.005 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [921121063-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.033 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [625420747-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b164fce at consistency null
2021-08-11 17:01:19.033 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [625420747-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.060 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [724714426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b137d85 at consistency null
2021-08-11 17:01:19.060 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [724714426-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.113 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [392580052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@503757c6 at consistency null
2021-08-11 17:01:19.113 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [392580052-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.166 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [411516438-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@773a04d8 at consistency null
2021-08-11 17:01:19.166 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [411516438-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.218 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [76694972-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@eab4dce at consistency null
2021-08-11 17:01:19.219 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [76694972-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.321 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1032124679-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4e94e6cb at consistency null
2021-08-11 17:01:19.321 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1032124679-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.423 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [404303240-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@247711b8 at consistency null
2021-08-11 17:01:19.424 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [404303240-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.526 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [961529619-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34cdd2a at consistency null
2021-08-11 17:01:19.526 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [961529619-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:19.730 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [795265571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@57dd35b0 at consistency null
2021-08-11 17:01:19.731 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [795265571-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:20.133 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [892355081-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3e16333e at consistency null
2021-08-11 17:01:20.133 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [892355081-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:20.936 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1679055946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@22f00f95 at consistency null
2021-08-11 17:01:20.936 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1679055946-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:21.738 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [597981316-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47b0336c at consistency null
2021-08-11 17:01:21.739 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [597981316-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:21.740 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:00:55.947Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:01:26.747 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [982157569-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79fefe9f at consistency null
2021-08-11 17:01:26.748 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [982157569-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:26.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [220050515-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@767ebd5d at consistency null
2021-08-11 17:01:26.847 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [220050515-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:26.900 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1261260726-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54d098a0 at consistency null
2021-08-11 17:01:26.900 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1261260726-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:26.952 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1271754255-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@123dea05 at consistency null
2021-08-11 17:01:26.952 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1271754255-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:27.055 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [635512054-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1cb9f4db at consistency null
2021-08-11 17:01:27.056 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [635512054-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:27.158 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1170121525-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fae3644 at consistency null
2021-08-11 17:01:27.158 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1170121525-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:27.361 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1682900171-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f6ba889 at consistency null
2021-08-11 17:01:27.361 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1682900171-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:27.763 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1154659163-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@327b0ea3 at consistency null
2021-08-11 17:01:27.764 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1154659163-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:28.566 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1756952995-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61b73b1b at consistency null
2021-08-11 17:01:28.567 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1756952995-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:29.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1487886474-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@f452d7f at consistency null
2021-08-11 17:01:29.370 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1487886474-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:29.371 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:00:55.947Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:01:29.989 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:01:29.991 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:01:30.664 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:30.666 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-45, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:01:30.667 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-45, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:01:30.667 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-45, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:30.667 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-45, inFlight=0, closed=true] closing connection
2021-08-11 17:01:30.667 DEBUG 10 --- [uster-worker-17] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:01:30.667 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 17:01:30.667 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 17:01:30.986 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:01:30.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-47, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:01:30.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-47, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:01:30.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-47, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:01:30.989 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-47, inFlight=0, closed=true] closing connection
2021-08-11 17:01:30.989 DEBUG 10 --- [uster-worker-16] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:01:30.990 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 17:01:30.990 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 17:01:34.374 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1146779144-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d78e3fe at consistency null
2021-08-11 17:01:34.375 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1146779144-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:34.427 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [356736435-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@779cf8e4 at consistency null
2021-08-11 17:01:34.427 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [356736435-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:34.480 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [983157593-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2fab6aef at consistency null
2021-08-11 17:01:34.480 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [983157593-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:34.585 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [407945096-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9a384b0 at consistency null
2021-08-11 17:01:34.585 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [407945096-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:34.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [254632124-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@531ee169 at consistency null
2021-08-11 17:01:34.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [254632124-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:34.789 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [9181743-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@43556836 at consistency null
2021-08-11 17:01:34.789 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [9181743-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:34.992 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1950075492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@375d8ea1 at consistency null
2021-08-11 17:01:34.992 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1950075492-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:35.395 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1762297748-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@224e1afe at consistency null
2021-08-11 17:01:35.395 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1762297748-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:36.197 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [231088777-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ffa277 at consistency null
2021-08-11 17:01:36.198 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [231088777-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:37.000 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1836780668-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6e037e46 at consistency null
2021-08-11 17:01:37.000 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1836780668-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:37.002 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:00:55.947Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:01:42.005 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1194951517-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3570dda1 at consistency null
2021-08-11 17:01:42.005 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1194951517-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:42.033 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2050136585-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67bba81b at consistency null
2021-08-11 17:01:42.033 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2050136585-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:42.085 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1180446124-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1c2c5281 at consistency null
2021-08-11 17:01:42.085 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1180446124-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:42.188 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1945223671-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69a8ee0c at consistency null
2021-08-11 17:01:42.188 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1945223671-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:42.291 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2019133375-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a86ab4d at consistency null
2021-08-11 17:01:42.291 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2019133375-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:42.394 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [374712431-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@60c73b76 at consistency null
2021-08-11 17:01:42.394 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [374712431-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:42.497 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [826529348-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e01d5b5 at consistency null
2021-08-11 17:01:42.497 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [826529348-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:42.699 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1279285218-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@30f780f7 at consistency null
2021-08-11 17:01:42.699 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1279285218-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:43.102 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2014336426-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5fc49967 at consistency null
2021-08-11 17:01:43.102 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2014336426-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:43.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1136495639-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6811562 at consistency null
2021-08-11 17:01:43.505 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1136495639-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:44.308 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1818461138-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@163dd47a at consistency null
2021-08-11 17:01:44.309 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1818461138-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:45.113 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [575813371-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a410e72 at consistency null
2021-08-11 17:01:45.113 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [575813371-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:45.916 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1486199326-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61b5a3fc at consistency null
2021-08-11 17:01:45.916 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1486199326-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:01:45.917 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:00:55.947Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:01:59.063 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.184.131:9042, scheduling delivery
2021-08-11 17:01:59.063 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event UP
2021-08-11 17:02:00.062 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.110.137:9042, scheduling delivery
2021-08-11 17:02:00.062 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event UP
2021-08-11 17:02:01.064 DEBUG 10 --- [uster-worker-18] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.242.184.131:9042
2021-08-11 17:02:01.064 DEBUG 10 --- [uster-worker-19] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on cassandra/10.242.110.137:9042
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-18] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-19] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-18] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-19] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-19] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-18] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-19] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:02:01.069 DEBUG 10 --- [uster-worker-18] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:02:01.073 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 17:02:01.073 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 17:02:09.965 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 17:02:10.165 DEBUG 10 --- [uster-worker-19] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 17:02:10.166 DEBUG 10 --- [uster-worker-19] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 17:02:10.166 DEBUG 10 --- [uster-worker-19] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 17:02:12.454 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 17:02:12.561 DEBUG 10 --- [uster-worker-18] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 17:02:12.561 DEBUG 10 --- [uster-worker-18] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 17:02:12.561 DEBUG 10 --- [uster-worker-18] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 17:02:31.072 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:02:31.073 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:02:41.078 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:02:41.079 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:02:51.262 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:02:51.265 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:03:01.095 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:03:01.096 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:03:11.087 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:03:11.089 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:03:31.111 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:03:31.114 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:03:41.347 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:03:41.349 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:04:01.115 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:04:01.117 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:04:21.451 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:04:21.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:04:31.118 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:04:31.120 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:04:36.483 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:04:36.486 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:04:51.477 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:04:51.479 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:05:01.147 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:05:01.150 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:05:31.178 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:05:31.180 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:05:41.577 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:05:41.579 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:06:01.205 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:06:01.207 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:06:11.595 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:06:11.597 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:06:16.675 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:06:16.678 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:06:31.225 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:06:31.226 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:06:46.858 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:06:46.859 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:07:01.233 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:07:01.235 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:07:16.885 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:07:16.887 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:07:31.244 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:07:31.246 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:07:47.081 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:07:47.084 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:08:01.274 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:08:01.276 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:08:31.289 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:08:31.291 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:08:42.184 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:08:42.186 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:08:57.249 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:08:57.253 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:08:59.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 17:08:59.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 17:08:59.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 17:08:59.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-46, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:08:59.251 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-46, inFlight=0, closed=true] closing connection
2021-08-11 17:08:59.252 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 17:08:59.253 DEBUG 10 --- [uster-worker-20] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 17:08:59.253 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 17:08:59.271 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 17:08:59.271 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 17:08:59.972 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 17:08:59.972 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 17:08:59.972 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 17:08:59.972 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:08:59.972 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-48, inFlight=0, closed=true] closing connection
2021-08-11 17:08:59.973 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 17:08:59.973 DEBUG 10 --- [uster-worker-21] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 17:08:59.973 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 17:08:59.983 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.110.137:9042, scheduling delivery
2021-08-11 17:08:59.983 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event DOWN
2021-08-11 17:09:00.253 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:00.256 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-47, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:09:00.257 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-47, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:09:00.257 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-47, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:00.257 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-47, inFlight=0, closed=true] closing connection
2021-08-11 17:09:00.257 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:09:00.257 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 17:09:00.257 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 17:09:00.973 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:00.976 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-49, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:09:00.976 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-49, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:09:00.976 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-49, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:00.976 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-49, inFlight=0, closed=true] closing connection
2021-08-11 17:09:00.977 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:09:00.977 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 17:09:00.977 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 17:09:00.983 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:09:00.984 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:09:02.258 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:02.259 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-48, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:09:02.260 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-48, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:09:02.260 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-48, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:02.260 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-48, inFlight=0, closed=true] closing connection
2021-08-11 17:09:02.260 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:09:02.260 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 17:09:02.260 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 17:09:02.353 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1752401120-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5b91380f at consistency null
2021-08-11 17:09:02.353 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1752401120-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:02.381 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [989429498-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6cd3201e at consistency null
2021-08-11 17:09:02.381 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [989429498-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:02.433 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [899302815-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@28485f6 at consistency null
2021-08-11 17:09:02.433 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [899302815-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:02.536 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1163122859-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4870640c at consistency null
2021-08-11 17:09:02.536 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1163122859-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:02.639 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [829624421-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a737c4d at consistency null
2021-08-11 17:09:02.639 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [829624421-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:02.841 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1082959391-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6f557ccd at consistency null
2021-08-11 17:09:02.841 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1082959391-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:02.977 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:02.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-50, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:09:02.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-50, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:09:02.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-50, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:02.980 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-50, inFlight=0, closed=true] closing connection
2021-08-11 17:09:02.980 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:09:02.981 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 17:09:02.981 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 17:09:03.044 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [660448556-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4958ffe4 at consistency null
2021-08-11 17:09:03.044 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [660448556-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:03.447 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2089859052-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2c3a2c7 at consistency null
2021-08-11 17:09:03.447 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2089859052-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:04.249 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1430876195-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3d337c6 at consistency null
2021-08-11 17:09:04.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1430876195-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:05.853 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1892718795-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@47249065 at consistency null
2021-08-11 17:09:05.853 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1892718795-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:05.855 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:08:56.846Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:09:06.260 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:06.263 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-49, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:09:06.263 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-49, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:09:06.263 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-49, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:06.263 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-49, inFlight=0, closed=true] closing connection
2021-08-11 17:09:06.264 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:09:06.264 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 17:09:06.264 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 17:09:06.981 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:06.983 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-51, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:09:06.983 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-51, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:09:06.983 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-51, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:06.983 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:09:06.983 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-51, inFlight=0, closed=true] closing connection
2021-08-11 17:09:06.984 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 17:09:06.984 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 17:09:10.859 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [980297528-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4cb0d6cf at consistency null
2021-08-11 17:09:10.859 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [980297528-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:10.886 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1136183374-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2f74d084 at consistency null
2021-08-11 17:09:10.886 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1136183374-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:10.913 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1567514738-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9dafde0 at consistency null
2021-08-11 17:09:10.914 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1567514738-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:10.941 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [697089492-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62408b9b at consistency null
2021-08-11 17:09:10.941 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [697089492-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:10.994 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1645221946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@280c2fb2 at consistency null
2021-08-11 17:09:10.994 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1645221946-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:11.096 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1735349902-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@21f52ac7 at consistency null
2021-08-11 17:09:11.097 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1735349902-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:11.199 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1227675617-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7f1ee6e4 at consistency null
2021-08-11 17:09:11.199 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1227675617-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:11.302 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [92451085-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1964b6cf at consistency null
2021-08-11 17:09:11.302 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [92451085-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:11.405 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1739696207-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@505c6bee at consistency null
2021-08-11 17:09:11.405 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1739696207-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:11.507 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2063026449-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5877c030 at consistency null
2021-08-11 17:09:11.508 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2063026449-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:11.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1439369841-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5056241c at consistency null
2021-08-11 17:09:11.710 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1439369841-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:12.112 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1171374876-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6c383217 at consistency null
2021-08-11 17:09:12.112 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1171374876-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:12.915 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [793311289-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5f083765 at consistency null
2021-08-11 17:09:12.915 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [793311289-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:14.264 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:14.266 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-50, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:09:14.266 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-50, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:09:14.266 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-50, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:14.267 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-50, inFlight=0, closed=true] closing connection
2021-08-11 17:09:14.267 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:09:14.267 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 17:09:14.267 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 17:09:14.518 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1865913088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@39981af4 at consistency null
2021-08-11 17:09:14.518 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1865913088-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:14.519 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:08:56.846Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:09:14.984 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:14.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-52, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:09:14.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-52, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:09:14.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-52, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:14.986 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-52, inFlight=0, closed=true] closing connection
2021-08-11 17:09:14.986 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:09:14.986 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 17:09:14.987 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 17:09:19.522 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1646429205-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@580faf1f at consistency null
2021-08-11 17:09:19.522 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1646429205-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:19.549 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [698380910-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@758a9683 at consistency null
2021-08-11 17:09:19.549 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [698380910-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:19.601 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1103224403-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6ba72448 at consistency null
2021-08-11 17:09:19.602 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1103224403-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:19.654 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [314436344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1211dc72 at consistency null
2021-08-11 17:09:19.654 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [314436344-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:19.706 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1682560662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@37c3ee78 at consistency null
2021-08-11 17:09:19.706 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1682560662-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:19.808 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1322314611-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@165c588d at consistency null
2021-08-11 17:09:19.809 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1322314611-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:20.011 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [28084156-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@448bab6a at consistency null
2021-08-11 17:09:20.011 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [28084156-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:20.214 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2084263771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@58650dc5 at consistency null
2021-08-11 17:09:20.214 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2084263771-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:20.617 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1458361132-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@70fd889f at consistency null
2021-08-11 17:09:20.617 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1458361132-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:21.019 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [523121137-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4113e3d at consistency null
2021-08-11 17:09:21.020 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [523121137-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:21.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2052773071-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1d58052e at consistency null
2021-08-11 17:09:21.422 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2052773071-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:22.225 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1560139088-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6da32865 at consistency null
2021-08-11 17:09:22.225 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1560139088-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:22.226 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:08:56.846Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:09:27.247 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1815615568-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7106fc3e at consistency null
2021-08-11 17:09:27.247 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1815615568-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:27.347 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [41360799-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7cebd4f5 at consistency null
2021-08-11 17:09:27.347 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [41360799-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:27.449 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [774327730-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2147cf33 at consistency null
2021-08-11 17:09:27.450 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [774327730-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:27.652 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [482499187-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@55532b7e at consistency null
2021-08-11 17:09:27.652 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [482499187-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:27.854 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [806632227-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1372dae6 at consistency null
2021-08-11 17:09:27.854 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [806632227-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:28.256 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [171262811-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@319ce801 at consistency null
2021-08-11 17:09:28.256 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [171262811-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:28.658 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [518767986-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2ebd9756 at consistency null
2021-08-11 17:09:28.658 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [518767986-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:29.460 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [284431646-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62f95124 at consistency null
2021-08-11 17:09:29.461 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [284431646-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:29.995 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:09:29.996 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:09:30.263 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [27851008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@67d90277 at consistency null
2021-08-11 17:09:30.263 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [27851008-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:30.267 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:30.270 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-51, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:09:30.270 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-51, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:09:30.270 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-51, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:30.270 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-51, inFlight=0, closed=true] closing connection
2021-08-11 17:09:30.270 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:09:30.270 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 17:09:30.270 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 17:09:30.987 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:30.989 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-53, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:09:30.989 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-53, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:09:30.989 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-53, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:09:30.989 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-53, inFlight=0, closed=true] closing connection
2021-08-11 17:09:30.989 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:09:30.990 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 17:09:30.990 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 17:09:31.070 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [564616693-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@742b60a5 at consistency null
2021-08-11 17:09:31.071 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [564616693-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:31.072 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:08:56.846Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:09:36.077 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1630002406-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c43e417 at consistency null
2021-08-11 17:09:36.078 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1630002406-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:36.125 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1372289337-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f37fc0d at consistency null
2021-08-11 17:09:36.125 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1372289337-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:36.178 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [393079545-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1b0b1a85 at consistency null
2021-08-11 17:09:36.178 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [393079545-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:36.247 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [213796116-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@582b26af at consistency null
2021-08-11 17:09:36.247 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [213796116-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:36.350 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1403086985-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@78dc6e33 at consistency null
2021-08-11 17:09:36.350 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1403086985-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:36.553 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1504297333-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1fe61e73 at consistency null
2021-08-11 17:09:36.553 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1504297333-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:36.756 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [560710299-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5c7ae038 at consistency null
2021-08-11 17:09:36.756 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [560710299-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:37.159 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1594078535-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d65bb46 at consistency null
2021-08-11 17:09:37.159 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1594078535-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:37.562 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [121531150-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4653fb2f at consistency null
2021-08-11 17:09:37.562 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [121531150-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:37.966 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [269197557-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69e96010 at consistency null
2021-08-11 17:09:37.966 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [269197557-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:38.769 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1863047029-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@62992635 at consistency null
2021-08-11 17:09:38.769 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1863047029-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:39.571 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [761327198-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12d60eab at consistency null
2021-08-11 17:09:39.572 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [761327198-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:39.574 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:08:56.846Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:09:44.578 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1509070567-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4b85055d at consistency null
2021-08-11 17:09:44.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1509070567-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:44.607 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [338921764-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@69ce15dd at consistency null
2021-08-11 17:09:44.607 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [338921764-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:44.635 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [624051368-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@424d3e5f at consistency null
2021-08-11 17:09:44.635 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [624051368-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:44.688 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [252337230-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@604cc3eb at consistency null
2021-08-11 17:09:44.688 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [252337230-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:44.791 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1107470054-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3009d52 at consistency null
2021-08-11 17:09:44.791 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1107470054-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:44.994 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1257270571-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a790012 at consistency null
2021-08-11 17:09:44.994 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1257270571-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:45.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2110814169-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@31ed7ac4 at consistency null
2021-08-11 17:09:45.397 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2110814169-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:45.800 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [551926785-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3738b489 at consistency null
2021-08-11 17:09:45.800 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [551926785-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:09:55.456 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.184.131:9042, scheduling delivery
2021-08-11 17:09:55.456 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event UP
2021-08-11 17:09:56.457 DEBUG 10 --- [uster-worker-20] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.242.184.131:9042
2021-08-11 17:09:56.461 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 17:09:56.462 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 17:09:56.462 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 17:09:56.462 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:56.465 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 17:09:58.462 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.110.137:9042, scheduling delivery
2021-08-11 17:09:58.462 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event UP
2021-08-11 17:09:59.463 DEBUG 10 --- [uster-worker-21] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on cassandra/10.242.110.137:9042
2021-08-11 17:09:59.465 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 17:09:59.465 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 17:09:59.465 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 17:09:59.465 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:09:59.467 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 17:09:59.549 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 17:09:59.756 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 17:09:59.757 DEBUG 10 --- [uster-worker-20] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 17:09:59.757 DEBUG 10 --- [uster-worker-20] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 17:10:03.367 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 17:10:03.971 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 17:10:03.971 DEBUG 10 --- [uster-worker-21] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 17:10:03.971 DEBUG 10 --- [uster-worker-21] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 17:10:29.468 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:10:29.470 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:10:33.973 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:10:34.021 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:10:59.472 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:10:59.473 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:11:26.416 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:11:26.418 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:11:29.476 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:11:29.477 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:11:56.426 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:11:56.429 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:11:59.479 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:11:59.481 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:12:01.457 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:12:01.460 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:12:26.476 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:12:26.478 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:12:29.483 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:12:29.484 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:12:59.485 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:12:59.486 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:13:11.589 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:13:11.591 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:13:21.596 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:13:21.598 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:13:29.488 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:13:29.489 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:13:59.491 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:13:59.493 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:14:06.713 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:14:06.717 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:14:21.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:14:21.755 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:14:29.495 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:14:29.497 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:14:41.783 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:14:41.784 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:14:59.497 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:14:59.499 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:15:29.500 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:15:29.501 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:15:36.990 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=1, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:15:36.991 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=1, closed=false] heartbeat query succeeded
2021-08-11 17:15:59.508 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:15:59.509 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:16:22.066 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:16:22.068 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:16:29.515 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:16:29.516 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:16:52.095 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:16:52.161 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:16:59.353 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 17:16:59.353 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] because: [/10.242.184.131:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 17:16:59.354 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preventing new connections for the next 1000 ms
2021-08-11 17:16:59.354 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-52, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:16:59.355 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-52, inFlight=0, closed=true] closing connection
2021-08-11 17:16:59.355 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host DOWN
2021-08-11 17:16:59.356 DEBUG 10 --- [uster-worker-22] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 17:16:59.357 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] starting reconnection attempt
2021-08-11 17:16:59.517 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:16:59.519 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:16:59.565 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.184.131:9042, scheduling delivery
2021-08-11 17:16:59.565 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event DOWN
2021-08-11 17:16:59.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] connection error
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:192)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253)
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133)
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:748)
2021-08-11 17:16:59.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Unexpected exception triggered (java.io.IOException: Connection reset by peer)
2021-08-11 17:16:59.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preventing new connections for the next 1000 ms
2021-08-11 17:16:59.778 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:16:59.779 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-54, inFlight=0, closed=true] closing connection
2021-08-11 17:16:59.779 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host DOWN
2021-08-11 17:16:59.779 DEBUG 10 --- [uster-worker-23] c.d.d.core.AbstractReconnectionHandler : First reconnection scheduled in 1000ms
2021-08-11 17:16:59.780 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] starting reconnection attempt
2021-08-11 17:16:59.877 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT DOWN /10.242.110.137:9042, scheduling delivery
2021-08-11 17:16:59.877 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event DOWN
2021-08-11 17:17:00.357 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:00.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-53, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:17:00.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-53, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:17:00.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-53, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:00.361 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-53, inFlight=0, closed=true] closing connection
2021-08-11 17:17:00.361 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:17:00.362 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 17:17:00.362 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 2000 ms
2021-08-11 17:17:00.780 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:00.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-55, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:17:00.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-55, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:17:00.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-55, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:00.783 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-55, inFlight=0, closed=true] closing connection
2021-08-11 17:17:00.783 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:17:00.784 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 2000 milliseconds
2021-08-11 17:17:00.784 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 2000 ms
2021-08-11 17:17:00.878 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:17:00.878 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:17:02.250 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1082578749-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@627ebaa9 at consistency null
2021-08-11 17:17:02.251 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1082578749-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:02.304 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [386596558-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@381fd778 at consistency null
2021-08-11 17:17:02.304 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [386596558-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:02.362 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:02.365 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-54, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:17:02.365 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-54, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:17:02.365 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-54, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:02.365 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-54, inFlight=0, closed=true] closing connection
2021-08-11 17:17:02.365 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:17:02.366 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 17:17:02.366 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 4000 ms
2021-08-11 17:17:02.407 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2032850763-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4af1b00e at consistency null
2021-08-11 17:17:02.407 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2032850763-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:02.609 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [956123771-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@501788ac at consistency null
2021-08-11 17:17:02.610 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [956123771-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:02.784 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:02.786 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-56, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:17:02.787 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-56, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:17:02.787 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-56, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:02.787 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-56, inFlight=0, closed=true] closing connection
2021-08-11 17:17:02.787 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 4000 milliseconds
2021-08-11 17:17:02.787 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:17:02.787 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 4000 ms
2021-08-11 17:17:02.813 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1320443238-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46ffc198 at consistency null
2021-08-11 17:17:02.813 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1320443238-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:03.215 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [243807984-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2a416303 at consistency null
2021-08-11 17:17:03.215 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [243807984-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:03.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [557263077-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@332ce1cd at consistency null
2021-08-11 17:17:03.618 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [557263077-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:04.021 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1942687786-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@79acb55f at consistency null
2021-08-11 17:17:04.021 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1942687786-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:04.424 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [536357976-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@416758f3 at consistency null
2021-08-11 17:17:04.424 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [536357976-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:04.826 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1351132897-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@601f62a7 at consistency null
2021-08-11 17:17:04.827 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1351132897-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:05.629 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1214307349-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@34e9d5e0 at consistency null
2021-08-11 17:17:05.629 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1214307349-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:05.631 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:16:56.662Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:17:06.366 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:06.368 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-55, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:17:06.368 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-55, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:17:06.368 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-55, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:06.368 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-55, inFlight=0, closed=true] closing connection
2021-08-11 17:17:06.368 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:17:06.368 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 17:17:06.369 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 8000 ms
2021-08-11 17:17:06.787 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:06.790 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-57, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:17:06.791 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-57, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:17:06.791 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-57, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:06.791 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-57, inFlight=0, closed=true] closing connection
2021-08-11 17:17:06.791 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:17:06.791 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 8000 milliseconds
2021-08-11 17:17:06.791 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 8000 ms
2021-08-11 17:17:10.635 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1661063027-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2873986f at consistency null
2021-08-11 17:17:10.635 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1661063027-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:10.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1027974567-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5328ed52 at consistency null
2021-08-11 17:17:10.687 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1027974567-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:10.740 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [686797632-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4839b7ca at consistency null
2021-08-11 17:17:10.740 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [686797632-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:10.792 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1721844284-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@32052df8 at consistency null
2021-08-11 17:17:10.792 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1721844284-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:10.895 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [188117964-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6817b37 at consistency null
2021-08-11 17:17:10.895 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [188117964-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:10.997 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [458131344-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@12d744c8 at consistency null
2021-08-11 17:17:10.998 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [458131344-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:11.200 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1315848714-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@17d526bf at consistency null
2021-08-11 17:17:11.200 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1315848714-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:11.603 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [573725017-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@ca4b350 at consistency null
2021-08-11 17:17:11.603 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [573725017-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:12.406 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [438499008-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1f13930 at consistency null
2021-08-11 17:17:12.406 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [438499008-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:13.208 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1967095827-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@29a8dcc2 at consistency null
2021-08-11 17:17:13.208 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1967095827-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:13.209 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:16:56.662Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:17:14.369 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:14.371 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-56, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:17:14.371 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-56, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:17:14.371 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-56, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:14.371 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-56, inFlight=0, closed=true] closing connection
2021-08-11 17:17:14.371 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:17:14.371 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 17:17:14.371 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 16000 ms
2021-08-11 17:17:14.792 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:14.793 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-58, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:17:14.794 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-58, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:17:14.794 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-58, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:14.794 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-58, inFlight=0, closed=true] closing connection
2021-08-11 17:17:14.794 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:17:14.794 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 16000 milliseconds
2021-08-11 17:17:14.794 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 16000 ms
2021-08-11 17:17:18.213 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1232986796-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@36d17e4e at consistency null
2021-08-11 17:17:18.213 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1232986796-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:18.265 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2041938507-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@59f0ea9a at consistency null
2021-08-11 17:17:18.265 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2041938507-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:18.368 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1949101899-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@791aec65 at consistency null
2021-08-11 17:17:18.368 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1949101899-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:18.470 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2107295429-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@480a726d at consistency null
2021-08-11 17:17:18.470 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [2107295429-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:18.673 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1532244824-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7514462c at consistency null
2021-08-11 17:17:18.673 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1532244824-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:19.075 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [237389219-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@2beec62e at consistency null
2021-08-11 17:17:19.075 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [237389219-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:19.478 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [493073819-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5396ef2 at consistency null
2021-08-11 17:17:19.478 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [493073819-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:19.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [920970268-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1e3bed22 at consistency null
2021-08-11 17:17:19.880 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [920970268-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:20.683 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1288753003-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4949fdef at consistency null
2021-08-11 17:17:20.683 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1288753003-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:21.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1885835279-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@144b8d5e at consistency null
2021-08-11 17:17:21.486 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1885835279-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:21.487 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:16:56.662Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:17:26.551 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1881681141-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5acb3fdb at consistency null
2021-08-11 17:17:26.551 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1881681141-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:26.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1635640040-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1bd88b5d at consistency null
2021-08-11 17:17:26.579 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1635640040-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:26.647 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [723305019-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1df8bb4c at consistency null
2021-08-11 17:17:26.647 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [723305019-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:26.700 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1731709119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5591e4d0 at consistency null
2021-08-11 17:17:26.700 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1731709119-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:26.752 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1447632680-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@56273c5e at consistency null
2021-08-11 17:17:26.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1447632680-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:26.855 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1479396779-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9fac7a4 at consistency null
2021-08-11 17:17:26.856 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1479396779-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:26.958 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [367903119-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@51002f0c at consistency null
2021-08-11 17:17:26.958 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [367903119-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:27.161 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1288077757-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@b6df698 at consistency null
2021-08-11 17:17:27.161 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1288077757-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:27.364 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1390962514-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6447ee67 at consistency null
2021-08-11 17:17:27.364 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1390962514-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:27.566 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1679730737-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@3bdfc2ac at consistency null
2021-08-11 17:17:27.566 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1679730737-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:27.769 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [528809092-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@6bf7f294 at consistency null
2021-08-11 17:17:27.769 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [528809092-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:28.171 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1731681696-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fca1044 at consistency null
2021-08-11 17:17:28.171 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1731681696-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:28.573 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1987343587-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@54a5e30c at consistency null
2021-08-11 17:17:28.574 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1987343587-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:29.376 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1740248380-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@11aa9196 at consistency null
2021-08-11 17:17:29.376 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1740248380-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:29.377 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:16:56.662Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:17:29.878 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:17:29.879 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:17:30.372 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:30.374 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-57, inFlight=0, closed=false] Error connecting to /10.242.184.131:9042 (Connection refused: /10.242.184.131:9042)
2021-08-11 17:17:30.374 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : Defuncting Connection[/10.242.184.131:9042-57, inFlight=0, closed=false] because: [/10.242.184.131:9042] Cannot connect
2021-08-11 17:17:30.374 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-57, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:30.374 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-57, inFlight=0, closed=true] closing connection
2021-08-11 17:17:30.374 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host /10.242.184.131:9042
2021-08-11 17:17:30.375 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Cluster : Failed reconnection to /10.242.184.131:9042 ([/10.242.184.131:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 17:17:30.375 DEBUG 10 --- [-reconnection-1] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] next reconnection attempt in 32000 ms
2021-08-11 17:17:30.794 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:30.797 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-59, inFlight=0, closed=false] Error connecting to cassandra/10.242.110.137:9042 (Connection refused: cassandra/10.242.110.137:9042)
2021-08-11 17:17:30.797 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : Defuncting Connection[cassandra/10.242.110.137:9042-59, inFlight=0, closed=false] because: [cassandra/10.242.110.137:9042] Cannot connect
2021-08-11 17:17:30.797 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-59, inFlight=0, closed=false] failed, remaining = 0
2021-08-11 17:17:30.797 DEBUG 10 --- [er-nio-worker-1] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-59, inFlight=0, closed=true] closing connection
2021-08-11 17:17:30.797 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Aborting onDown because a reconnection is running on DOWN host cassandra/10.242.110.137:9042
2021-08-11 17:17:30.797 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Cluster : Failed reconnection to cassandra/10.242.110.137:9042 ([cassandra/10.242.110.137:9042] Cannot connect), scheduling retry in 32000 milliseconds
2021-08-11 17:17:30.797 DEBUG 10 --- [-reconnection-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] next reconnection attempt in 32000 ms
2021-08-11 17:17:34.380 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1608689043-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@451da265 at consistency null
2021-08-11 17:17:34.381 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1608689043-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.408 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [48572818-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1a8ac2b2 at consistency null
2021-08-11 17:17:34.408 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [48572818-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.435 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1466140225-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@61ccff6c at consistency null
2021-08-11 17:17:34.435 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1466140225-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.487 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1623685732-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@24afe4e2 at consistency null
2021-08-11 17:17:34.488 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1623685732-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.546 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [709068242-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@c897498 at consistency null
2021-08-11 17:17:34.546 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [709068242-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.598 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [21074097-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@405aa8e9 at consistency null
2021-08-11 17:17:34.598 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [21074097-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.650 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1255890774-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@46eb1811 at consistency null
2021-08-11 17:17:34.651 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1255890774-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [294918195-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@1db86cda at consistency null
2021-08-11 17:17:34.753 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [294918195-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:34.955 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [667902662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5a52e641 at consistency null
2021-08-11 17:17:34.956 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [667902662-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:35.358 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [713472662-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@4f5a1002 at consistency null
2021-08-11 17:17:35.358 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [713472662-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:36.165 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [976707967-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@516fa268 at consistency null
2021-08-11 17:17:36.165 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [976707967-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:36.968 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [198423434-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@fefca23 at consistency null
2021-08-11 17:17:36.968 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [198423434-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:36.970 WARN 10 --- [pool-8-thread-1] o.j.diskstorage.log.kcvs.KCVSLog : Could not read messages for timestamp [2021-08-11T17:16:56.662Z] (this read will be retried)
org.janusgraph.core.JanusGraphException: Could not execute operation due to backend exception
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:57)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:159)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller.run(KCVSLog.java:732)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Could not successfully complete backend operation due to repeated temporary exceptions after PT4S
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:101)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:55)
... 9 common frames omitted
Caused by: org.janusgraph.diskstorage.TemporaryBackendException: Temporary failure in storage backend
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$null$2(CQLKeyColumnValueStore.java:126)
at io.vavr.API$Match$Case0.apply(API.java:3174)
at io.vavr.API$Match.of(API.java:3137)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.lambda$static$3(CQLKeyColumnValueStore.java:123)
at io.vavr.control.Try.getOrElseThrow(Try.java:671)
at org.janusgraph.diskstorage.cql.CQLKeyColumnValueStore.getSlice(CQLKeyColumnValueStore.java:290)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:798)
at org.janusgraph.diskstorage.log.kcvs.KCVSLog$MessagePuller$1.call(KCVSLog.java:795)
at org.janusgraph.diskstorage.util.BackendOperation.execute(BackendOperation.java:148)
at org.janusgraph.diskstorage.util.BackendOperation$1.call(BackendOperation.java:162)
at org.janusgraph.diskstorage.util.BackendOperation.executeDirect(BackendOperation.java:69)
... 10 common frames omitted
Caused by: java.util.concurrent.ExecutionException: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at io.vavr.control.Try.of(Try.java:62)
at io.vavr.concurrent.FutureImpl.lambda$run$2(FutureImpl.java:199)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
... 3 common frames omitted
Caused by: com.datastax.driver.core.exceptions.NoHostAvailableException: All host(s) tried for query failed (tried: /10.242.154.72:9042 (com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)))
at com.datastax.driver.core.RequestHandler.reportNoMoreHosts(RequestHandler.java:265)
at com.datastax.driver.core.RequestHandler.access$1200(RequestHandler.java:62)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.findNextHostAndQuery(RequestHandler.java:357)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.retry(RequestHandler.java:539)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.processRetryDecision(RequestHandler.java:521)
at com.datastax.driver.core.RequestHandler$SpeculativeExecution.onSet(RequestHandler.java:791)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1233)
at com.datastax.driver.core.Connection$Dispatcher.channelRead0(Connection.java:1151)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:99)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.timeout.IdleStateHandler.channelRead(IdleStateHandler.java:287)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.MessageToMessageDecoder.channelRead(MessageToMessageDecoder.java:102)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.handler.codec.ByteToMessageDecoder.fireChannelRead(ByteToMessageDecoder.java:321)
at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:295)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.ChannelInboundHandlerAdapter.channelRead(ChannelInboundHandlerAdapter.java:93)
at com.datastax.driver.core.InboundTrafficMeter.channelRead(InboundTrafficMeter.java:38)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:355)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:377)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:363)
at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919)
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163)
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714)
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650)
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576)
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
... 1 common frames omitted
2021-08-11 17:17:41.974 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1465264484-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@42424571 at consistency null
2021-08-11 17:17:41.974 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1465264484-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:42.027 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1218145042-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5d07873b at consistency null
2021-08-11 17:17:42.027 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1218145042-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:42.081 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1227278311-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@71571211 at consistency null
2021-08-11 17:17:42.081 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1227278311-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:42.184 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1602187946-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@9870ca at consistency null
2021-08-11 17:17:42.184 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1602187946-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:42.386 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [956595496-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@5021d465 at consistency null
2021-08-11 17:17:42.387 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [956595496-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:42.789 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [710169313-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@317eaf05 at consistency null
2021-08-11 17:17:42.789 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [710169313-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:43.591 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1437774259-0] Doing retry 1 for query com.datastax.driver.core.BoundStatement@7870e4a1 at consistency null
2021-08-11 17:17:43.591 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.RequestHandler : [1437774259-0] Error querying /10.242.154.72:9042 : com.datastax.driver.core.exceptions.UnavailableException: Not enough replicas available for query at consistency LOCAL_QUORUM (2 required but only 1 alive)
2021-08-11 17:17:56.952 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.184.131:9042, scheduling delivery
2021-08-11 17:17:56.952 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] received event UP
2021-08-11 17:17:57.953 DEBUG 10 --- [uster-worker-23] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on /10.242.184.131:9042
2021-08-11 17:17:57.959 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] marking host UP
2021-08-11 17:17:57.959 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 17:17:57.959 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node /10.242.184.131:9042
2021-08-11 17:17:57.959 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:57.962 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 17:17:58.366 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Cluster : Received event EVENT UP /10.242.110.137:9042, scheduling delivery
2021-08-11 17:17:58.367 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Host.STATES : [/10.242.110.137:9042] received event UP
2021-08-11 17:17:59.367 DEBUG 10 --- [uster-worker-22] c.d.driver.core.ControlConnection : [Control connection] Refreshing node info on cassandra/10.242.110.137:9042
2021-08-11 17:17:59.370 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] marking host UP
2021-08-11 17:17:59.370 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Cancelling reconnection attempt since node is UP
2021-08-11 17:17:59.370 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Cluster : Preparing 54 prepared queries on newly up node cassandra/10.242.110.137:9042
2021-08-11 17:17:59.370 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 1 new connections, total = 1
2021-08-11 17:17:59.371 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-60, inFlight=0, closed=false] Connection established, initializing transport
2021-08-11 17:18:00.853 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 17:18:01.249 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Host.STATES : [/10.242.184.131:9042] preparing to open 0 new connections, total = 1
2021-08-11 17:18:01.249 DEBUG 10 --- [uster-worker-23] c.d.driver.core.HostConnectionPool : Created connection pool to host /10.242.184.131:9042 (1 connections needed, 1 successfully opened)
2021-08-11 17:18:01.249 DEBUG 10 --- [uster-worker-23] com.datastax.driver.core.Session : Added connection pool for /10.242.184.131:9042
2021-08-11 17:18:11.264 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] Connection[cassandra/10.242.110.137:9042-60, inFlight=0, closed=false] Transport initialized, connection ready
2021-08-11 17:18:11.870 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Host.STATES : [cassandra/10.242.110.137:9042] preparing to open 0 new connections, total = 1
2021-08-11 17:18:11.871 DEBUG 10 --- [uster-worker-22] c.d.driver.core.HostConnectionPool : Created connection pool to host cassandra/10.242.110.137:9042 (1 connections needed, 1 successfully opened)
2021-08-11 17:18:11.871 DEBUG 10 --- [uster-worker-22] com.datastax.driver.core.Session : Added connection pool for cassandra/10.242.110.137:9042
2021-08-11 17:18:29.371 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:18:29.373 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:18:59.402 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:18:59.404 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:19:05.285 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:19:05.287 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:19:29.429 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:19:29.459 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:19:35.311 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:19:35.314 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:19:55.332 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:19:55.346 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:19:59.460 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:19:59.461 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:20:15.353 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:20:15.355 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:20:25.363 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:20:25.365 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:20:29.477 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:20:29.479 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:20:55.384 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:20:55.446 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:20:59.484 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:20:59.486 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:21:20.453 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:21:20.455 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:21:25.455 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:21:25.457 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:21:29.503 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:21:29.505 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:21:59.523 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:21:59.525 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:22:00.494 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:22:00.496 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:22:29.554 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:22:29.556 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:22:59.574 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:22:59.575 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:23:15.621 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:23:15.623 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-1, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:23:29.605 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:23:29.607 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:23:59.627 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:23:59.629 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:24:15.870 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-60, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:24:15.872 DEBUG 10 --- [er-nio-worker-0] com.datastax.driver.core.Connection : Connection[cassandra/10.242.110.137:9042-60, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:24:29.643 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:24:29.645 DEBUG 10 --- [er-nio-worker-3] com.datastax.driver.core.Connection : Connection[/10.242.154.72:9042-32, inFlight=0, closed=false] heartbeat query succeeded
2021-08-11 17:24:40.901 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] was inactive for 30 seconds, sending heartbeat
2021-08-11 17:24:40.904 DEBUG 10 --- [er-nio-worker-2] com.datastax.driver.core.Connection : Connection[/10.242.184.131:9042-58, inFlight=0, closed=false] heartbeat query succeeded